When I read Harlan's entry it made me laugh because it's so true. Too often are the times when a sysadmin claims they didn't do anything and then during an analysis you see something that looks like this for the access time listing in a directory:
2008-Feb-25 22:50:56.043375 UTC
2008-Feb-25 22:50:56.074625 UTC
2008-Feb-25 22:51:01.012125 UTC
2008-Feb-25 22:51:01.184000 UTC
2008-Feb-25 22:51:01.418375 UTC
2008-Feb-25 22:51:01.434000 UTC
2008-Feb-25 22:51:01.465250 UTC
2008-Feb-25 22:51:01.496500 UTC
2008-Feb-25 22:51:01.559000 UTC
2008-Feb-25 22:51:01.590250 UTC
2008-Feb-25 22:51:01.621500 UTC
2008-Feb-25 22:51:01.652750 UTC
2008-Feb-25 22:51:01.684000 UTC
2008-Feb-25 22:51:01.715250 UTC
2008-Feb-25 22:51:01.746500 UTC
2008-Feb-25 22:51:01.871500 UTC
2008-Feb-25 22:51:01.902750 UTC
2008-Feb-25 22:51:01.934000 UTC
2008-Feb-25 22:51:01.949625 UTC
2008-Feb-25 22:51:01.965250 UTC
2008-Feb-25 22:51:01.996500 UTC
2008-Feb-25 22:51:02.027750 UTC
2008-Feb-25 22:51:06.699820 UTC
2008-Feb-25 22:51:06.824838 UTC
2008-Feb-25 22:51:06.824838 UTC
2008-Feb-25 22:51:06.887348 UTC
2008-Feb-25 22:51:06.918602 UTC
2008-Feb-25 22:51:06.981111 UTC
2008-Feb-25 22:51:07.012366 UTC
2008-Feb-25 22:51:07.074875 UTC
2008-Feb-25 22:51:12.138120 UTC
2008-Feb-25 22:51:12.200630 UTC
2008-Feb-25 22:51:12.200630 UTC
2008-Feb-25 22:51:12.278766 UTC
2008-Feb-25 22:51:12.372530 UTC
2008-Feb-25 22:51:12.388157 UTC
2008-Feb-25 22:51:12.435039 UTC
When I see something like this I start to think of a card game called "bull****" where one person tells a lie and the other players get to try to call them on the lie. This is also where I like to pull out the "sysadmin stick" and like texas ranger in Talladega nights I say "One of you turds is about to get smacked in the mouth".
You see, when a first responder or sysadmin approaches a system to "investigate" they seem to like to do some combination of the following:
Run task manager
Run netstat
Run a rootkit detector
Run an antivirus scan
Delete files that look like they are related to something bad
Run a backup/restore job
Defrag the hard drive - usually because someone said the computer was slow
Not that I've never seen these things before...but rightfully I can't just point at sysadmins. External consultants are just as guilty. During one response I got a call and arrived shortly thereafter only to find listings similar to the above, but over the entire system and on multiple systems. I looked over at the manager and said "what did they do?". He gave me a sheet that detailed the actions of the consultants (wow they actually logged it) and low and behold they updated AV definitions and scanned each system with a full scan from symantec AV corporate edition. Granted, these people are just doing their jobs but if you have a large organization you must absolutely make sure the external consultants used are aware of and adhere to the company incident response procedures/policy. You must also make certain that you train the people within the organization that are commonly the first responders.
*thwack*
Monday, February 25, 2008
Subscribe to:
Post Comments (Atom)
3 comments:
My biggest take-away from this post is the last sentence. Too often, when we get a call, a great deal of activity has taken place on the systems already, much of which obfuscates or completely eradicates what most would deem as "evidence".
Most IT staff has very little idea of what to do when an incident occurs. To make matters worse, when an incident *does* occur, there is usually widespread panic, starting at the top, due to potential risks and exposure to the organization. Most senior management does not understand that it does, in fact, require thought and training even to perform first response...most IT staffs don't do it everyday, so they need to be trained to respond in an appropriate manner.
Without guidance from the top, preferably in the form of an incident response plan, the IT staff are often left to fend for themselves.
Even if they know what they're doing, they're often under pressure to "just get it fixed". Taking the time to properly analyze the situation might not be appreciated.
That bull**** often rolls down hill and the easiest way not to get covered in it is to just get the system back and move on.
Very true, many "well intended" sysadmins just want to fix the problem. In doing so, they often cause the root problem to be masked by the actions they have performed.
Post a Comment