Thoughts on incident response

Richard Bejtlich was blogging about thoughts on incident response and asked that people respond in their own blogs...so here goes..


Thoughts:

The first 24 hours are crucial. Kind of a cliche but oh so true. If the wrong decision is made early, you can and will end up in a bad spot as the incident progresses.

Sacrifice Speed for GOOD DOCUMENTATION. Speed, while important should not replace your documentation. Remember..if it isn't written down, it didn't happen. Slow down and make sure your notes are up to snuff. I don't know how many times I've had to refer to my notes post incident..but without them I'd be lost.

The post incident follow up is just as important as the incident response effort. Always make sure you do this.
Some common questions:
What went right?
What went wrong?
What could have been done better?
Have all of the new security recommendations been implemented? If not, why?
Does the client require assistance with implementation?
What lessons have been learned?
How many man hours did the incident require?
Was any data lost during the response?
Was any hardware damaged?
Did the response procedures work?
Were the response procedures followed?
Did containment methods work?
Were there any tools that could have helped?
How soon after detection was the response initiated?
Was communication effective?
How long was the business operation of the department negatively impacted?
Was there adequate cooperation between the client Technical and administrative staff and the IRT?
Was the incident properly reported as per policy?

This phase also presents the IRT with the opportunity to review its own internal process and procedures for response efforts.

Items to review internally:
Were forensically sound procedures followed?
Was a lead handler assigned?
Were all communication channels used? Were the proper parties contacted?
Was the incident properly documented?
Do the procedures need to change?



Lessons Learned:

1) Obtain authority and determine the incident owner early. If you are not authorized to control a scene then what in the world are you doing there? Getting authorized to do XYZ will help get you past a lot of obstacles. I provide people with a form letter for just this purpose.

2) Prepare Emergency systems in advance. In one major incident, I ended up rebuilding a network infrastructure, including deployment of: switches, DHCP, NSM(debian running snort connected to a teeny tap), and had to rip apart two rather humorous looking switch closets. I now have images of a few types of systems ready and on hand.

3) You can either not sleep or not eat, but not both. On the same incident I was awake and on site for the first 48 hours, then got about 3 hours of sleep, because of the critical state of the incident. This incident took about 700 man hours to resolve. If you choose not to eat or sleep you'll run out of gas early in the incident and be unable to perform when you're needed most.

4) The war room is important. Make sure if your incident handling efforts take a while, you have adequate space on site that is a secure base of operations.

5) Get management involved. Until management starts to see the sometimes superhuman effort required to handle incidents, they will not understand the need to devote time and money to Incident Preparation.