Monday, February 25, 2008

The sysadmin stick

When I read Harlan's entry it made me laugh because it's so true. Too often are the times when a sysadmin claims they didn't do anything and then during an analysis you see something that looks like this for the access time listing in a directory:

2008-Feb-25 22:50:56.043375 UTC
2008-Feb-25 22:50:56.074625 UTC
2008-Feb-25 22:51:01.012125 UTC
2008-Feb-25 22:51:01.184000 UTC
2008-Feb-25 22:51:01.418375 UTC
2008-Feb-25 22:51:01.434000 UTC
2008-Feb-25 22:51:01.465250 UTC
2008-Feb-25 22:51:01.496500 UTC
2008-Feb-25 22:51:01.559000 UTC
2008-Feb-25 22:51:01.590250 UTC
2008-Feb-25 22:51:01.621500 UTC
2008-Feb-25 22:51:01.652750 UTC
2008-Feb-25 22:51:01.684000 UTC
2008-Feb-25 22:51:01.715250 UTC
2008-Feb-25 22:51:01.746500 UTC
2008-Feb-25 22:51:01.871500 UTC
2008-Feb-25 22:51:01.902750 UTC
2008-Feb-25 22:51:01.934000 UTC
2008-Feb-25 22:51:01.949625 UTC
2008-Feb-25 22:51:01.965250 UTC
2008-Feb-25 22:51:01.996500 UTC
2008-Feb-25 22:51:02.027750 UTC
2008-Feb-25 22:51:06.699820 UTC
2008-Feb-25 22:51:06.824838 UTC
2008-Feb-25 22:51:06.824838 UTC
2008-Feb-25 22:51:06.887348 UTC
2008-Feb-25 22:51:06.918602 UTC
2008-Feb-25 22:51:06.981111 UTC
2008-Feb-25 22:51:07.012366 UTC
2008-Feb-25 22:51:07.074875 UTC
2008-Feb-25 22:51:12.138120 UTC
2008-Feb-25 22:51:12.200630 UTC
2008-Feb-25 22:51:12.200630 UTC
2008-Feb-25 22:51:12.278766 UTC
2008-Feb-25 22:51:12.372530 UTC
2008-Feb-25 22:51:12.388157 UTC
2008-Feb-25 22:51:12.435039 UTC

When I see something like this I start to think of a card game called "bull****" where one person tells a lie and the other players get to try to call them on the lie. This is also where I like to pull out the "sysadmin stick" and like texas ranger in Talladega nights I say "One of you turds is about to get smacked in the mouth".

You see, when a first responder or sysadmin approaches a system to "investigate" they seem to like to do some combination of the following:
Run task manager
Run netstat
Run a rootkit detector
Run an antivirus scan
Delete files that look like they are related to something bad
Run a backup/restore job
Defrag the hard drive - usually because someone said the computer was slow

Not that I've never seen these things before...but rightfully I can't just point at sysadmins. External consultants are just as guilty. During one response I got a call and arrived shortly thereafter only to find listings similar to the above, but over the entire system and on multiple systems. I looked over at the manager and said "what did they do?". He gave me a sheet that detailed the actions of the consultants (wow they actually logged it) and low and behold they updated AV definitions and scanned each system with a full scan from symantec AV corporate edition. Granted, these people are just doing their jobs but if you have a large organization you must absolutely make sure the external consultants used are aware of and adhere to the company incident response procedures/policy. You must also make certain that you train the people within the organization that are commonly the first responders.


Impact analysis

I recently co-taught a class on using helix for incident response and forensics. During the course I found myself thinking back to past conversations and papers I've read on the subject of impact and impact analysis...

It's well known that we have an impact on a system, and it's perhaps even better known that we have the ability to be the single greatest force that gets exerted on a system while it's up and running.

For the purposes of this entry I am referring to the use of WFT and the monolithic configuration file that unnecessarily uses multiple tools that use the same API's and same function calls on helix. First, a quick discussion about this..if two tools use the same API and the same function calls to collect data, it's pretty safe to say that we will collect the same information from the execution of either tool barring some form of subversion or direct attack against a specific tool.

Since WFT in Helix (1.9) uses them I'll refer to: Dumpel and psloglist here.

Dumpel comes to us from the Windows 2000 resource kit. psloglist comes from sysinternals. Both tools have the ability to extract event logs from a system. Psloglist is definitely more feature rich but both accomplish the same thing when run locally against a system.

To do a comparison I used pe explorer and extracted the dependencies and PE import tables of each.

It's a little burdensome to read it all and post the complete diff here but here's a screenshot. Let's just say that the only differences in the dependency files are what I'm showing.

Ok, so the dependencies are the same. How about the function calls from each? The main functions called to open, read and close event logs for both tools comes out of advapi32.dll,kernel32.dll and user32.dll. They both use the exact same libraries (psloglist uses more because it's more feature rich) which is no real surprise.

After limited testing so far it seems that the tools will provide the same results. So my question is...why run both tools? Why the need to execute the same function calls from two tools?

If a large part of response is about minimizing impact (since doing no harm is not possible) then there is no good reason I can think of to run two tools that use the same API and function calls. We use more memory, and have a greater impact than is required on a system. More specifically there is a problem with helix's use of WFT and it's simply that it seems to treat every problem as a nail when in fact there is no need to run all of these extraneous tools that increase our impact when there is just no good technical need. This all goes to knowing your tools and understanding what they do and how they work.

What are your thoughts?

Sunday, February 17, 2008


I guess I've been blogging now for an entire year. I certainly have had some periods of absence but that's to be expected when you get busy or have an infant I suppose. If you read this with any frequency I hope the past year has inspired you to think a little more about the field and what you do as a practitioner, or maybe it's inspired you to block the blog so my ramblings don't hurt your eyes. Either way this has been an interesting experiment for me. I've been exposed to some interesting members of the field and had some really good conversations and debates - all things I would have missed out on had I not decided to give this a try. I originally started doing this as my way to contribute to the community of forensics and incident response, but during this time I've asked myself more than once "why do I bother?" and then I visit a forum or read a recent question on a mailing list and I have my answer. We are a young and growing field. Every day there are more people with more questions and each day some of those people are too nervous or cautious (for various reasons) to ask their question or present their ideas. In this dysfunctional field I honestly can't blame them either. New challenges arise every day and every day there is a new trick to learn, a new wrinkle to be gained. I bother to contribute in my own way because there are many who don't bother for whatever reason they may claim. If we wish to be considered a science then we must ask the questions that no one else will or those that we feel uncomfortable asking. This is how growth occurs, not in numbers of investigators and certified individuals but in challenging what we think we know and what others think as well. So, ask your questions and contribute in your own way, or ask me your questions and I'll ask them for you. I've also asked myself why am I still blogging? I have yet to discover why people read blogs, this one in particular. Is it for entertainment? Ideas, information, or keeping current? I have found that since I started the clustrmaps visitor tracking map(thanks Mark!) there are people that read this blog, some people even subscribe to it(Thanks!) yet I have received few comments. I sometimes get the sensation that the field is a one way street where few contribute and lots of people sit on the sidelines.

I found it to be incredibly interesting this past year when I was approached by a gentlemen who asked if the University he represented could syndicate some of my posts. Wow I thought, now that's cool. This honestly got me back in to deeper study of criminal justice and criminalistics. I managed to get my hands on some great books through my local libraries. I said at one point in the past year that I will forever be a student and that still holds true.

By far the best book I read all year was Hans Gross' Criminal Psychology. It was a tough read but his book was probably the "smartest" book I've read in a long time because it made me think and made me question a lot of things. If you are in the field, READ THIS BOOK. It's available as an E-book but you can buy it for about $25 or so.

There is one other book I hold high above the rest. It's relatively new but wow it's got so much in it. In a training I held recently I was asked which books I would recommend and I said "That's easy. Windows Forensic Analysis, in fact it's right here in my bag if you want to look at it." I held up the book and said "this is the best book on the market on the subject right now". I listed a few other books as well but they live on my bookshelf either at home or at work and I occasionally bring them back and forth. In my review of WFA I said it was going to sit on my quick grab shelf. That just hasn't been the case. It goes where I go.

Some principles from this past year:

What works today may not work tomorrow.

After an email discussion on incident preparation and defense in depth, I ran in to the person with whom I had been emailing and he said this: "As a Buddhist monk once told me..By the time you have to take a shit, it's too late to begin digging a hole".

A science can be better understood by studying other fields.

Cooperation is counter-operational.

I'm not sure what the future holds for me or this blog, all I know is there's a lot to be done and a lot that needs to be talked about.

Friday, February 15, 2008

CCE retained

I took a little time today and completed the recertification process in order to retain my CCE status. I had hoped for a more thorough practical but I realize how much effort it takes to put these things together.