How do incidents occur?

As someone who thrives in what can only be described as organized chaos I get to answer the question "How did this happen?". Naturally when incidents occur, people like to blame the person lowest on the totem pole, which typically ends up being the system administrator. It is arguable whether or not anyone can blame the sysadmin that has to manage sometimes over 70 servers and 400 desktop computers but I tend to look at the organization holistically before I place blame on the lowely admins. Just as there is 'defense in depth' there are multiple layers of failure. Let's just say that failure(incidents), like security begins at the top of the organization. Take your highest ranking three letter exec and ask what their views on security are. Odds are you'll get a "secu-what?" response out of them most likely followed by a "I have people that work for me that do that sort of thing". Next, visit your financial officer and ask them what they think of security and what your security budget is this coming year. You might get a "well it's included in the IT budget" or "Good question, I'll get back to you on that". Then ask your C[TI]O/CISO what their take on security is. You'll probably end up getting a long winded response about the CIA triad of security and how your organization takes security seriously. Ask them how much time they allocate for their sysadmins to focus on security of their systems. Does it fall in that Other Duties as assigned category? Now go visit HR to ask what policies are in place regarding the security of your organization. There are lots of good policies in place in a lot of organizations but many places just say "yeah we have a policy about that". If you get that response, ask who is directly responsible for enforcing it, then ask that person about the policy. Then ask how they educate their people about the policy.

Travel down the exec path until you reach the system administrator who looks at you like your absolutely crazy as they show you this mornings help desk tickets and say "I don't have time to deal with security. I do what I can but no one gives me enough time to focus on security, so I fit it in when I can". Ask the admin what kind of access controls he or she has in place. You might be surprised one way or the other. Then ask them how they detect incidents. Ever seen a deer in headlights? You just did. Now ask them how they respond to incidents. Time for another one of those.."good question, let me find out" responses. At this point you proceed to slap your forehead and wonder..."how did it take so long for an incident to occur here?"

Does this sound at all familiar to anyone?

How do incidents occur? It's commonly a series of failures in the organization. It's sometimes amusing to pull all of the people listed above in to an incident response meeting. The poor guy at the top is fretting over how much money is being lost, the other execs are just dumbfounded because there are policies in place, and they take this security thing seriously. Meanwhile the sysadmin is sitting quietly at the end of the table wondering if their resume is up to date.


Let me recap. Incidents, like security begin at the top of the organizational chart. If the head of your organization isn't serious about it, then no one else will have the resources they need and incidents will occur, and more often than you might like. Just remember this little fact. In sports, when the team fails, it's not the players that usually get fired (occassionally they'll get traded yes), it's the coach. The coach is the one responsible for managing the efforts of the team below him or her.