So you've hired an expert. They're supposed to be representing you as an 'expert' in a court of law. Here are some things you never want to hear them say. These are real statements. If anyone wonders why regulation of our field is a requirement, think about these quotes and what it means when a qualified expert makes them.
"The EnCE is like a little badge of honor".
"I took two five day courses, and no other formal training".
"My analysis looked the other direction".
"Most likely it's an issue related to our examination computer".
"We have no procedure for chain of custody".
Friday, January 30, 2009
Friday thoughts to expand on
I've been having some interesting discussions of late. Instead of keeping these things 'close to the chest' I figured I'd throw out some thoughts and questions(some rhetorical) here. This is more of a placeholder than anything but if you have thoughts of your own please share them.
Do particular sources of artifacts speak more loudly or carry more weight when they are reviewed by the trier of fact(think lay person in a decision making role) in a case?
Do more sources of artifacts automatically mean a stronger case, or must they agree to make a stronger case? Which ones matter and which ones have a "who cares" factor?
Think preponderance of evidence.
Reasonable belief is a game that can't be played in digital forensic science due to the circumstantial nature of digital evidence as it pertains to intrusions. It's like working a murder case with no body and no weapon. A clinical approach is required and a clinical approach requires criteria yet there has been no established criteria in any state.
If given SYSTEM or root level privileges on a computer, could you, with your knowledge set, defeat your own ability to accurately analyze that same system?
Does Access equal Acquired? Acquired is what the law looks for.
With operating systems and applications being as over-engineered as they currently are, is there any possibility of anyone being able to show cause and effect?
Do particular sources of artifacts speak more loudly or carry more weight when they are reviewed by the trier of fact(think lay person in a decision making role) in a case?
Do more sources of artifacts automatically mean a stronger case, or must they agree to make a stronger case? Which ones matter and which ones have a "who cares" factor?
Think preponderance of evidence.
Reasonable belief is a game that can't be played in digital forensic science due to the circumstantial nature of digital evidence as it pertains to intrusions. It's like working a murder case with no body and no weapon. A clinical approach is required and a clinical approach requires criteria yet there has been no established criteria in any state.
If given SYSTEM or root level privileges on a computer, could you, with your knowledge set, defeat your own ability to accurately analyze that same system?
Does Access equal Acquired? Acquired is what the law looks for.
With operating systems and applications being as over-engineered as they currently are, is there any possibility of anyone being able to show cause and effect?
Wednesday, January 28, 2009
Using RegRipper
I have been doing oodles of analysis lately (6+TB since December!) and have been making heavy, heavy use of registry analysis in each case. For this I've been using Regripper so I figured I would dedicate at least one post to it, and how I make use of it. I also happen to be a firm believer that someone can tell a person to use something 1000 times, but until they actually see it in use or can see the utility in using it, they are less likely to follow your recommendation. With that said, here's a few ways I use regripper.
1) Determine group membership.
Like no other tool I've used to date, the samparse module has saved me hours of analysis time in determining who belonged to what group and what privileges they had. Just this morning I saw some mailing list posts about determining why there was no user account listed for a particular account in the SAM on a windows XP box. The answer was rather simple - The account was a domain account.
Here's an abridged example of what I'm talking about:
User Information
------------------------------------------------
Username: Administrator [500]
Username: Guest [501]
Username: SUPPORT_388945a0 [1002]
Username: SUPPORT_3f151ab9 [1003]
Username: HelpAssistant [1004]
Username: ASPNET [1006]
Group membership information
------------------------------------------------
Group Name: Power Users [1]
Users: S-1-5-21-1461745249-492156796-3006755465-1119
Group Name: Administrators [3]
Users: S-1-5-21-296978250-731684933-3931576523-500
To explain, the top portion are the local user accounts. Under the Group membership section we can see the power users group with a SID/RID combination that doesn't fit with the system. How do I know this? This is easily determined by examination of the Administrators group section of the file. The Administrator account is a well known SID of S-1-5-N-500. As such, I now know the system SID and can say that the account in the power users group is NOT a local account. This can and *should* be correlated to the software hive analysis of the profilelist key and its values that details user account and the SID. When looking at the software hive, you can quickly determine the account name from the SID and the domain name in question.
2) Determine services installed.
Regripper has a great module to determine what services were on the system sorted by last write time. Comparing this to an exemplar list of windows services allows you to a) do data reduction and b) determine what services may have been installed or leveraged by an intruder.
3) Windows Firewall configuration.
I once had an incident where an external consultant claimed the firewall was not disabled by them(which is what caused the incident), however the firewall according to regripper was disabled, and other logs confirmed this with all roads pointing to the consultants.
4) Confirmation of devices in use on the system.
There's been a number of times when I could say to a customer that had been infected with removable media malware "You'll want to make sure you clean up these devices" or identify "rogue" devices that had been used on a system.
5) Determine network configuration.
In a world dominated by a lot of DHCP and variety of network usage, determining the last known IP address used by the system is invaluable, especially when you need to plug the IP address in to various network analysis utilities. I use this (and other markers) to make sure I've got the system I'm supposed to have.
6) Determine user activity.
Naturally there's a bunch of good things here to look at and every investigation includes a look here.
In short there's untold ways to use regripper for analysis. These are just a few small examples of how I've made use of the tool. I don't really know how many people are using it, but there sure are a bunch of people who aren't, and that's just a shame. It's a great tool that simplifies the process of registry analysis and simply stated it saves time. As we all know, time is money and saving both is important to everyone involved in investigations. Don't forget, as I showed in the video I posted some time ago, it can be run in concert with F-response to look at an otherwise "locked" registry hive on a remote system, while it's live.
There's much more I could say about using regripper but I imagine there will be plenty included in Windows Forensic Analysis second edition.
1) Determine group membership.
Like no other tool I've used to date, the samparse module has saved me hours of analysis time in determining who belonged to what group and what privileges they had. Just this morning I saw some mailing list posts about determining why there was no user account listed for a particular account in the SAM on a windows XP box. The answer was rather simple - The account was a domain account.
Here's an abridged example of what I'm talking about:
User Information
------------------------------------------------
Username: Administrator [500]
Username: Guest [501]
Username: SUPPORT_388945a0 [1002]
Username: SUPPORT_3f151ab9 [1003]
Username: HelpAssistant [1004]
Username: ASPNET [1006]
Group membership information
------------------------------------------------
Group Name: Power Users [1]
Users: S-1-5-21-1461745249-492156796-3006755465-1119
Group Name: Administrators [3]
Users: S-1-5-21-296978250-731684933-3931576523-500
To explain, the top portion are the local user accounts. Under the Group membership section we can see the power users group with a SID/RID combination that doesn't fit with the system. How do I know this? This is easily determined by examination of the Administrators group section of the file. The Administrator account is a well known SID of S-1-5-N-500. As such, I now know the system SID and can say that the account in the power users group is NOT a local account. This can and *should* be correlated to the software hive analysis of the profilelist key and its values that details user account and the SID. When looking at the software hive, you can quickly determine the account name from the SID and the domain name in question.
2) Determine services installed.
Regripper has a great module to determine what services were on the system sorted by last write time. Comparing this to an exemplar list of windows services allows you to a) do data reduction and b) determine what services may have been installed or leveraged by an intruder.
3) Windows Firewall configuration.
I once had an incident where an external consultant claimed the firewall was not disabled by them(which is what caused the incident), however the firewall according to regripper was disabled, and other logs confirmed this with all roads pointing to the consultants.
4) Confirmation of devices in use on the system.
There's been a number of times when I could say to a customer that had been infected with removable media malware "You'll want to make sure you clean up these devices" or identify "rogue" devices that had been used on a system.
5) Determine network configuration.
In a world dominated by a lot of DHCP and variety of network usage, determining the last known IP address used by the system is invaluable, especially when you need to plug the IP address in to various network analysis utilities. I use this (and other markers) to make sure I've got the system I'm supposed to have.
6) Determine user activity.
Naturally there's a bunch of good things here to look at and every investigation includes a look here.
In short there's untold ways to use regripper for analysis. These are just a few small examples of how I've made use of the tool. I don't really know how many people are using it, but there sure are a bunch of people who aren't, and that's just a shame. It's a great tool that simplifies the process of registry analysis and simply stated it saves time. As we all know, time is money and saving both is important to everyone involved in investigations. Don't forget, as I showed in the video I posted some time ago, it can be run in concert with F-response to look at an otherwise "locked" registry hive on a remote system, while it's live.
There's much more I could say about using regripper but I imagine there will be plenty included in Windows Forensic Analysis second edition.
Saturday, January 10, 2009
The internet is for porn
It's a well known saying to be sure. The internet is for porn. Perhaps the funniest depiction of this was a chappelle show episode . It's probably NSFW if you've never seen it, and it's absolutely hilarious, but it definitely raises a number of points that speak to the industry we work in.
Here's a scenario for you..
Joe the finance executive at a bank is browsing the web. He visits a news site, and a link to a site that suggest adult conversations is flashing in the ad banner space. Joe is happily married yet he's curious, and temptation overrules logical thought. He's acting completely right brained. Living in the moment for the moment, not thinking about the future. He visits the "adult conversation" site and bam! he's assaulted with pictures and popups of all forms of pornography. Now Joe's in a whole other world. His basal instincts have taken over and what was supposed to be a quick check in of the local news turned in to a trip down porn lane. A few clicks later and an install of flash player, and he's merrily watching some streaming porn on his laptop at work.
Joe is happy, Joe is enjoying himself.
You, sitting in your position of overwatch, looking for strange and outlandish network behavior notice Joe's computer doing something like this:
111.222.33.44,FSPA,27289,72.213.167.190,FSA,80,909,573,11,6,0,0,TCP,POST / HTTP/1.1..Host: iexujguw.com..Content-Length: 116..Connection: close.....,HTTP/1.1 200 OK..Server: nginx/0.5.33..Date: Fri. 05 June 2008 16:20:27 GMT.
111.222.33.44,FSPA,42583,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 16:40:16 GMT.
111.222.33.44,FSPA,16197,212.55.163.216,FSA,80,848,1054,11,7,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 17:00:17 GMT.
111.222.33.44,FSPA,3884,66.102.1.101,FSPA,80,1334,12549,13,14,0,0,TCP,POST /safebrowsing/downloads?client=navclient-auto-ffox&appver=3.0.5&pver=2.2&wr,HTTP/1.1 200 OK..Content-Type: application/vnd.google.safebrowsing-update..Date:,,1010,,
111.222.33.44,FSPA,4124,66.102.1.100,FSPA,80,1322,12549,13,14,0,0,TCP,POST /safebrowsing/downloads?client=navclient-auto-ffox&appver=3.0.5&pver=2.2&wr,HTTP/1.1 200 OK..Content-Type: application/vnd.google.safebrowsing-update..Date:,,1010,,
111.222.33.44,FSPA,59619,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 17:20:18 GMT.
111.222.33.44,FSPA,51889,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 17:40:18 GMT.
111.222.33.44,FSPA,4392,66.102.1.101,FSPA,80,1438,12549,13,14,0,0,TCP,POST /safebrowsing/downloads?client=navclient-auto-ffox&appver=3.0.5&pver=2.2&wr,HTTP/1.1 200 OK..Content-Type: application/vnd.google.safebrowsing-update..Date:,,1010,,
111.222.33.44,FSPA,58415,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 18:00:20 GMT.
Joe has managed to visit one of the countless porn sites that is actually owned and/or operated by a sub-group in organized crime, or hosts malicious flash or other malware.
Joe, in his quest for local news, and following his temptations has opened up the organization to a whole new world of risk.
Joe is compromised.
Not only is he compromised but he's managed to get a copy of Sinowal loaded on to his computer. Joe, being the finance director at the bank has access to all of the financial information of all of the bank's customers, and he uses this access to run reports. Joe is now responsible for exposing the records for all of the customers of the bank.
Ok, enough about Joe.
What I find interesting about this all is how in a matter of a few seconds, one can go from a nice clean site to an awful bodega of porn in a matter of a few clicks. Like six degrees of separation, the internet appears to be '6 clicks to porn', as in from any site you can end up at a porn site in 6 clicks. It's like walking down a street in a major city and from block to block, you can go from the best part of the city, to the worst and most dangerous. I don't know many people that would willingly walk down a dark dank avenue known to have muggers and other dangerous people. Yet, people do it daily on the internet. Most users don't seem to put the two together. For some reason it's as if people still believe that computers are in a separate reality and whatever happens on a computer does not have the ability to affect real people or their lives.
If the saying is to be believed, that computers are deterministic then it can easily be stated that computers don't do bad things. People using computers doing stupid things leads to computers doing bad or stupid things.
That said, in the case of Joe, do you think he should be punished or should you simply investigate the computer intrusion? Do your intrusion investigations lead to investigation of the people using the computer? Is Joe the Witness, the Perpetrator, or the Victim? What's your decision making process?
There is a bit of intel to be gained from this post. Sinowal has definite characteristics on the network. In my experiences they are as follows:
Here's a scenario for you..
Joe the finance executive at a bank is browsing the web. He visits a news site, and a link to a site that suggest adult conversations is flashing in the ad banner space. Joe is happily married yet he's curious, and temptation overrules logical thought. He's acting completely right brained. Living in the moment for the moment, not thinking about the future. He visits the "adult conversation" site and bam! he's assaulted with pictures and popups of all forms of pornography. Now Joe's in a whole other world. His basal instincts have taken over and what was supposed to be a quick check in of the local news turned in to a trip down porn lane. A few clicks later and an install of flash player, and he's merrily watching some streaming porn on his laptop at work.
Joe is happy, Joe is enjoying himself.
You, sitting in your position of overwatch, looking for strange and outlandish network behavior notice Joe's computer doing something like this:
111.222.33.44,FSPA,27289,72.213.167.190,FSA,80,909,573,11,6,0,0,TCP,POST / HTTP/1.1..Host: iexujguw.com..Content-Length: 116..Connection: close.....,HTTP/1.1 200 OK..Server: nginx/0.5.33..Date: Fri. 05 June 2008 16:20:27 GMT.
111.222.33.44,FSPA,42583,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 16:40:16 GMT.
111.222.33.44,FSPA,16197,212.55.163.216,FSA,80,848,1054,11,7,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 17:00:17 GMT.
111.222.33.44,FSPA,3884,66.102.1.101,FSPA,80,1334,12549,13,14,0,0,TCP,POST /safebrowsing/downloads?client=navclient-auto-ffox&appver=3.0.5&pver=2.2&wr,HTTP/1.1 200 OK..Content-Type: application/vnd.google.safebrowsing-update..Date:,,1010,,
111.222.33.44,FSPA,4124,66.102.1.100,FSPA,80,1322,12549,13,14,0,0,TCP,POST /safebrowsing/downloads?client=navclient-auto-ffox&appver=3.0.5&pver=2.2&wr,HTTP/1.1 200 OK..Content-Type: application/vnd.google.safebrowsing-update..Date:,,1010,,
111.222.33.44,FSPA,59619,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 17:20:18 GMT.
111.222.33.44,FSPA,51889,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 17:40:18 GMT.
111.222.33.44,FSPA,4392,66.102.1.101,FSPA,80,1438,12549,13,14,0,0,TCP,POST /safebrowsing/downloads?client=navclient-auto-ffox&appver=3.0.5&pver=2.2&wr,HTTP/1.1 200 OK..Content-Type: application/vnd.google.safebrowsing-update..Date:,,1010,,
111.222.33.44,FSPA,58415,212.55.163.216,FSA,80,784,687,10,6,0,0,TCP,POST /4D3D07E3ABDFC3C5/qxUX4xETUFYBWqc0kaWCzvoCcAQCYSNwZgcyFiBAByC73XXm0CcAYgVSB,HTTP/1.1 200 OK..Server: nginx/0.5.35..Date: Fri. 05 June 2008 18:00:20 GMT.
Joe has managed to visit one of the countless porn sites that is actually owned and/or operated by a sub-group in organized crime, or hosts malicious flash or other malware.
Joe, in his quest for local news, and following his temptations has opened up the organization to a whole new world of risk.
Joe is compromised.
Not only is he compromised but he's managed to get a copy of Sinowal loaded on to his computer. Joe, being the finance director at the bank has access to all of the financial information of all of the bank's customers, and he uses this access to run reports. Joe is now responsible for exposing the records for all of the customers of the bank.
Ok, enough about Joe.
What I find interesting about this all is how in a matter of a few seconds, one can go from a nice clean site to an awful bodega of porn in a matter of a few clicks. Like six degrees of separation, the internet appears to be '6 clicks to porn', as in from any site you can end up at a porn site in 6 clicks. It's like walking down a street in a major city and from block to block, you can go from the best part of the city, to the worst and most dangerous. I don't know many people that would willingly walk down a dark dank avenue known to have muggers and other dangerous people. Yet, people do it daily on the internet. Most users don't seem to put the two together. For some reason it's as if people still believe that computers are in a separate reality and whatever happens on a computer does not have the ability to affect real people or their lives.
If the saying is to be believed, that computers are deterministic then it can easily be stated that computers don't do bad things. People using computers doing stupid things leads to computers doing bad or stupid things.
That said, in the case of Joe, do you think he should be punished or should you simply investigate the computer intrusion? Do your intrusion investigations lead to investigation of the people using the computer? Is Joe the Witness, the Perpetrator, or the Victim? What's your decision making process?
There is a bit of intel to be gained from this post. Sinowal has definite characteristics on the network. In my experiences they are as follows:
- Always communicates with nginx webservers, acting as proxies.
- Uses a 20 minute timer, and will skew on occasion but not by much.
- Uses a static HTTP POST with 16 hex characters followed by a trailing slash.
- Uses a domain generation routine much like Srizbi and will do an HTTP POST to / at that domain name.
Sunday, January 4, 2009
A call to arms
Well not really to arms in that we should be brandishing weapons. Rather we should be brandishing our legal arms and need to stand up for ourselves and protect our industry and our right to perform our duties. Nothing in 2008 illustrated this more than the Private Investigator laws that popped up in many states. The legislators are perhaps the worst informed about how our industry actually works. Conversely they have been lobbied the hardest by those who wish to force their wills upon us and our industry. In some states, forensic analysts and small business owners stood up for themselves and managed to have laws drawn up appropriately. However, in many other states we stood idle while the laws that affect us were drawn up around us. We still have no governing body for digital forensics. We are still an industry born out of Information Technology rather than science. Yes, steps are being taken to get us included in scientific forensic organizations, but this is not enough. We need to get more involved in the decisions that affect us the most. For example, physical memory analysis has been huge since it's true inception in 2005. We need to decide how we want this type of analysis to be used in a legal setting. There will likely come a time when legal matters can be decided on the basis of memory analysis. Many people were taken by surprise when the PI laws seemingly materialized from thin air, yet they had been in process for quite some time already.
We need to be prepared as an industry. We need to stand up for our industry. We need to improve our industry. We need to pay attention to the legislation that affects our ability to perform our duties. Digital Forensics must become Digital Forensic Science.
We need to be prepared as an industry. We need to stand up for our industry. We need to improve our industry. We need to pay attention to the legislation that affects our ability to perform our duties. Digital Forensics must become Digital Forensic Science.
Saturday, January 3, 2009
The golden hour
I briefly mentioned the golden hour in a recent post. Matt beat me to the punch in his post, but I wanted to spend a little time on the golden hour. In my opinion, the golden hour exists in two places for incidents and incident response.
1) The time between the time of compromise and time of containment
2) The time between the time of containment and the point in time where response truly begins
This post will focus on the second frame of time.
The golden hour is a medical term generally reserved for a period of time that is most critical for a patient in need of care. The idea is that if the proper care can be given within this golden hour, the chances the patient will survive increases dramatically. This isn't necessarily a hard period of 60 minutes, rather it can be seen as a principle of providing high quality care as rapidly as possible.
I like to think of it as a guiding principle for Incident Response.
You see, I think it applies to Incident Response in a way that I don't think many people pay attention to. Of all the people I have been in contact with, there is one that really gets it ( actually he preaches it as well ) and I can't count the times Harlan has said something to the effect of "incident responders are like EMT's". EMT's play a huge role in the success of the golden hour. They are responsible for providing immediate care and transport of the patient. The symbol of the EMT is known as the star of life. It's a symbol many have seen but I wonder how many people understand the meaning behind each point and the symbol in general.
I won't attempt to reiterate the details of the symbol and it's humble beginnings, and under no circumstance would I attempt to in any way take anything away from the folks that wear the symbol by doing a direct comparison between EMS work and Incident Response. I know several people that wear or have worn that symbol and simply stated, they save lives. We work with bits and bytes and there is no comparison.
That said, as I have said before, the best way to master our own field is to study the methods used in other fields. So here goes.
The 6 points of the 'star of life' each represent a specific portion of the role EMS plays. They are as follows:
If we can do the following, our OODA loops close quicker, our response is more effective and who knows..we might be able to get closer to the truth. Our goals are (or should be):
Early detection I think I covered already.
Early Reporting
The reason this is here is twofold. We want our clients and customers to file reports with us as early as possible. In addition, we need to be able to provide our reports, to the proper people in a timely manner and communicate as often and as appropriately as possible. The worst decisions are made when the decision maker is uninformed. It is our duty to inform the decision maker as early as possible provided we have something substantial to share.
Take the following in to consideration regarding reporting:
This is where Matt's post filled in a gap for me. Rapid Response is essential to effective response. It is for this reason that a mandatory response time should be established. Every minute that passes after an incident occurs without a response is a minute that you've given to the intruder. Unfortunately responding to an incident takes time. Typically there's a huge gap whereby the intruder can do any number of things to the system.
Getting on scene physically or remotely as quickly as possible is tough. I've had incidents where I had to get to no fewer than 5 physical locations. There are others who have geographical concerns. This is where F-response closes the gap and allows us to respond rapidly. Think about it. What is in the standard toolbox of a technician that might respond to a compromised system? Antivirus programs, file deletion, backup/restore software, hijackthis(or similar), netstat, task manager and other similar diagnostic tools. These tools are known to stomp all over the things we tend to care about. By the time we get on scene a technician will likely have touched the computer. Again, this is where F-reponse can play a role. I'd love to get F-response in to the hands of every technician out there. Why F-response over something like Encase Enterprise you might ask? Simple.
1) EE is expensive. Think of what else you could do with that money that would beef up your response.
2) F-response is inexpensive. This is the most appropriately priced tool in the industry.
3) It's simple. I can give a client a CD and tell them to simply pop it in a drive, and I can begin to analyze a system. I can teach them how to use the product in about 5 minutes.
4) It's tool agnostic. It provides the ability to use any of your other tools to do analysis. You aren't locked in to Encase or FTK or X-ways, you can use all three if you want to.
Putting a tool this simple in the hands of technicians would allow for an even more granular tier or response whereby the technician could gain access to a system in a protected manner and they could then perform basic triage functions. Antivirus scans could be run safely to identify any malware the AV product is capable of detecting. The rebuilding process could begin on another physical computer while you either travel or prepare to respond remotely. Either way what I think we tend to miss most often is setting the expectation for the technician or first responder. What do you want them to do, and what do you not want them to do? I can't count the times I've heard techs say "Sure you can have the disk for imaging but I need to get some files off first", or "I copied the files I could find and zipped them up, then I deleted the files. I thought I was doing the right thing". From their perspective I can understand that, however that's not what we want them to do. In order the create a granular response capability whereby the technician or local sysadmin can assist what do you need?
1) Training
2) Usable Procedures - think basic flowchart
2) Simple tools that work
In addition to tools like F-response, live response procedures must exist and they must be followed. If your team members are going out in the field, and they are dealing with systems that are still on when they arrive, they need to know how to conduct a live response. Where resources are spread thin, you can provide a rapid response capability by creating tiers of response. Not all incidents require a full incident response team traveling on site. Some incidents are minor or can be triaged by local staff before you can arrive.
Rapid response is founded on early and accurate reporting. Imagine getting on scene and not having the proper equipment. That would be a disaster. We need to be informed before we respond. You should be able to define for someone what you need to know.
When you are on site in a response situation, you should be able to make decisions when you need to. There are times when you should follow the playbook and there are times when you should throw out the playbook because the playbook doesn't cover the situation at hand and you need to make a solid tactical decision.
Some things to consider about rapid response:
Once we arrive on scene we need to know what we're doing and we need to be able to do it well. Getting on scene and screwing something up is unacceptable. The right person needs to be available for the response and the right people need to be available from the organization you're serving. It does us no good to have the web guy when we need the database guy. So what helps when you're on scene?
This for incident responders is relatively simple. Not only must we travel safely, but if we have taken a disk or a disk image we need to transport it safely. Carrying a bare disk around is not something I enjoy seeing. Drives should be packaged carefully as should systems that are seized. If a drive dies in our possession we are to held responsible. So what must we do?
We need to prevent tampering, static and shock from affecting a drive we've seized. These are the principles of device and drive seizure yet they are not always taken in to account. I use a combination of drive cases and anti-static bags for this purpose.
Take the following in to consideration for care in transit:
Finally we need to provide solid forensic analysis. Once the drive or image has arrived in our various evidence lockers and we are ready to perform an analysis, it must be thorough, complete, accurate and precise. The use of "I believe" or "perhaps" in our conclusions is nothing more than self reassurance and these statements tend to falter in the face of scrutiny. We must be careful in our analysis and in our presentation of data. I will not spend to much time on this as it tends to be a subject worthy of other future discussions. Forensic analysis is the final destination for Incident Response.
Our goal is to reach this stage as rapidly and as effectively as possible. Our clients count on it. It is our duty as incident responders to provide proper response in the golden hour. Have any thoughts to add?
1) The time between the time of compromise and time of containment
2) The time between the time of containment and the point in time where response truly begins
This post will focus on the second frame of time.
The golden hour is a medical term generally reserved for a period of time that is most critical for a patient in need of care. The idea is that if the proper care can be given within this golden hour, the chances the patient will survive increases dramatically. This isn't necessarily a hard period of 60 minutes, rather it can be seen as a principle of providing high quality care as rapidly as possible.
I like to think of it as a guiding principle for Incident Response.
You see, I think it applies to Incident Response in a way that I don't think many people pay attention to. Of all the people I have been in contact with, there is one that really gets it ( actually he preaches it as well ) and I can't count the times Harlan has said something to the effect of "incident responders are like EMT's". EMT's play a huge role in the success of the golden hour. They are responsible for providing immediate care and transport of the patient. The symbol of the EMT is known as the star of life. It's a symbol many have seen but I wonder how many people understand the meaning behind each point and the symbol in general.
I won't attempt to reiterate the details of the symbol and it's humble beginnings, and under no circumstance would I attempt to in any way take anything away from the folks that wear the symbol by doing a direct comparison between EMS work and Incident Response. I know several people that wear or have worn that symbol and simply stated, they save lives. We work with bits and bytes and there is no comparison.
That said, as I have said before, the best way to master our own field is to study the methods used in other fields. So here goes.
The 6 points of the 'star of life' each represent a specific portion of the role EMS plays. They are as follows:
- Detection
- Reporting
- Response
- On Scene Care
- Care in Transit
- Transfer to Definitive Care
If we can do the following, our OODA loops close quicker, our response is more effective and who knows..we might be able to get closer to the truth. Our goals are (or should be):
- Early Detection
- Early Reporting
- Rapid Response
- Good on scene practices
- Care in transit
- Solid Forensic analysis
Early detection I think I covered already.
Early Reporting
The reason this is here is twofold. We want our clients and customers to file reports with us as early as possible. In addition, we need to be able to provide our reports, to the proper people in a timely manner and communicate as often and as appropriately as possible. The worst decisions are made when the decision maker is uninformed. It is our duty to inform the decision maker as early as possible provided we have something substantial to share.
Take the following in to consideration regarding reporting:
- Provide your clients with a standard reporting template. This template should be updated regularly to ensure it allows the client to provide you with enougn information to respond properly
- Provide multiple intake options such as Web, phone, paper
- Use a standardized report template
- Notify the proper people within your organization within 24 hours of an incident occurring
- Get in touch with the client as soon as possible to establish expectations
This is where Matt's post filled in a gap for me. Rapid Response is essential to effective response. It is for this reason that a mandatory response time should be established. Every minute that passes after an incident occurs without a response is a minute that you've given to the intruder. Unfortunately responding to an incident takes time. Typically there's a huge gap whereby the intruder can do any number of things to the system.
Getting on scene physically or remotely as quickly as possible is tough. I've had incidents where I had to get to no fewer than 5 physical locations. There are others who have geographical concerns. This is where F-response closes the gap and allows us to respond rapidly. Think about it. What is in the standard toolbox of a technician that might respond to a compromised system? Antivirus programs, file deletion, backup/restore software, hijackthis(or similar), netstat, task manager and other similar diagnostic tools. These tools are known to stomp all over the things we tend to care about. By the time we get on scene a technician will likely have touched the computer. Again, this is where F-reponse can play a role. I'd love to get F-response in to the hands of every technician out there. Why F-response over something like Encase Enterprise you might ask? Simple.
1) EE is expensive. Think of what else you could do with that money that would beef up your response.
2) F-response is inexpensive. This is the most appropriately priced tool in the industry.
3) It's simple. I can give a client a CD and tell them to simply pop it in a drive, and I can begin to analyze a system. I can teach them how to use the product in about 5 minutes.
4) It's tool agnostic. It provides the ability to use any of your other tools to do analysis. You aren't locked in to Encase or FTK or X-ways, you can use all three if you want to.
Putting a tool this simple in the hands of technicians would allow for an even more granular tier or response whereby the technician could gain access to a system in a protected manner and they could then perform basic triage functions. Antivirus scans could be run safely to identify any malware the AV product is capable of detecting. The rebuilding process could begin on another physical computer while you either travel or prepare to respond remotely. Either way what I think we tend to miss most often is setting the expectation for the technician or first responder. What do you want them to do, and what do you not want them to do? I can't count the times I've heard techs say "Sure you can have the disk for imaging but I need to get some files off first", or "I copied the files I could find and zipped them up, then I deleted the files. I thought I was doing the right thing". From their perspective I can understand that, however that's not what we want them to do. In order the create a granular response capability whereby the technician or local sysadmin can assist what do you need?
1) Training
2) Usable Procedures - think basic flowchart
2) Simple tools that work
In addition to tools like F-response, live response procedures must exist and they must be followed. If your team members are going out in the field, and they are dealing with systems that are still on when they arrive, they need to know how to conduct a live response. Where resources are spread thin, you can provide a rapid response capability by creating tiers of response. Not all incidents require a full incident response team traveling on site. Some incidents are minor or can be triaged by local staff before you can arrive.
Rapid response is founded on early and accurate reporting. Imagine getting on scene and not having the proper equipment. That would be a disaster. We need to be informed before we respond. You should be able to define for someone what you need to know.
When you are on site in a response situation, you should be able to make decisions when you need to. There are times when you should follow the playbook and there are times when you should throw out the playbook because the playbook doesn't cover the situation at hand and you need to make a solid tactical decision.
Some things to consider about rapid response:
- Geographic separation
- Resource availability and tiered response
- Training of response staff
- Training of first responders
- Tools that enhance or provide rapid response capability
- Knowledge of diverse set of operating systems
- Established protocols
- Response team is cleared to make decisions
- Response team should be properly informed
Once we arrive on scene we need to know what we're doing and we need to be able to do it well. Getting on scene and screwing something up is unacceptable. The right person needs to be available for the response and the right people need to be available from the organization you're serving. It does us no good to have the web guy when we need the database guy. So what helps when you're on scene?
- Establish protocols for common scenarios
- Ensure your tools are updated and work as expected
- Be thorough
- Be flexible
- Have the right people available
This for incident responders is relatively simple. Not only must we travel safely, but if we have taken a disk or a disk image we need to transport it safely. Carrying a bare disk around is not something I enjoy seeing. Drives should be packaged carefully as should systems that are seized. If a drive dies in our possession we are to held responsible. So what must we do?
We need to prevent tampering, static and shock from affecting a drive we've seized. These are the principles of device and drive seizure yet they are not always taken in to account. I use a combination of drive cases and anti-static bags for this purpose.
Take the following in to consideration for care in transit:
- Safety
- Shock
- Static
- Tampering
Finally we need to provide solid forensic analysis. Once the drive or image has arrived in our various evidence lockers and we are ready to perform an analysis, it must be thorough, complete, accurate and precise. The use of "I believe" or "perhaps" in our conclusions is nothing more than self reassurance and these statements tend to falter in the face of scrutiny. We must be careful in our analysis and in our presentation of data. I will not spend to much time on this as it tends to be a subject worthy of other future discussions. Forensic analysis is the final destination for Incident Response.
Our goal is to reach this stage as rapidly and as effectively as possible. Our clients count on it. It is our duty as incident responders to provide proper response in the golden hour. Have any thoughts to add?
Thursday, January 1, 2009
Tales from the field - the degausser
Until recently I had never seen nor used a degausser. Sure I'd read about them, heard folks talk about them, but never had the fun of using one or seeing one in use. The solution of choice had been to ship the drives out for shredding. With the economy being what it is we invested in a degausser that looks something like this
Like anyone else that gets a new toy we were excited to test this thing out. We had a few spare hard drives ready for destruction and wanted to see if degaussing even worked. The instructions were simple: put a drive in the tray, put the gloves on, and step on the peddle. Swirl the tray around in the provided circles for 10 seconds per side, and voila you'll have a cooked drive.
Sounded simple enough....
First we tried a 2.5" drive. BBBBBRRRRRRRRRRMMMMMMMMMMMMM...the degausser sounded off as it did its dirty work. Flip the drive over and BBBBBRRRRRRRRRRMMMMMMMMMMMMM...We pulled the drive off. Yep, it was toasty alright and it was accompanied by the the smell of hot metal. I took the drive back to my workstation after it had cooled off a bit. I wanted to see what had happened and wanted to see if the drive was still usable.
I plugged it in to my trusty writeblocker and turned on the writeblocker. The familiar whirring of the platters was quickly accompanied by a shocked "HOLY @$*%" coming out of my mouth. I could read the drive just fine. Not only was the drive intact, the filesystem was consistent and boy could I read the files. I quickly called over my teammates and we all looked at the drive contents as if we'd never seen a directory before.
Quickly I ran down the hall accompanied by another round of BBBBBRRRRRRRRRRMMMMMMMMMMMMM as the degausser chewed up another drive. This time it was a 3.5". I walked in to the room as the tray rattled around on top of the degausser. Not long after I entered the room was I walking back down the hall with two very hot hard drives in hand. I couldn't wait to see these not work.
I plugged in one drive and wouldn't you know it, I could read that one too. In disbelief I tried the other one. Yep I could read that one too. I pulled the drive from the write blocker and tried to format it. That actually did not work. Ok, so I could read the drive but could not write to it. That's reassuring but I care about someone being able to read it.
It was as if I'd been punched in the face. It was then that I recalled what the gentleman at EDR said when I was talking to him at technosecurity. He said "The problem with degaussers is that yeah sure, it's been marked as degaussed, but that's all you've got. Someone else saying they ran that drive through a degausser. What guarantee do you have that the drive is useless?"
Apparently I had no guarantee that the degausser works. If we couldn't get it to work properly, how would the people at the location actually housing this thing be able to ensure the drives were rendered useless. Promptly the vendor was contacted. They suggested we try to rotate the drive on another axis in addition to laying the drive flat. Willing to give it another try we turned the drive on its side and rotated it for the prescribed time. This time when I plugged in the drive, I could not read it. This was reassuring. Each successive time we tried a drive I was unable to read it.
This certainly reinforced the need to test all new equipment regardless of what its purpose is. It also reinforces the need to periodically verify the equipment to ensure its still working and you can trust it. Degaussing a drive is an interesting experience. I know that I for one would rather see a drive physically shredded, ground or smelted than degaussed. At least that way I can guarantee the drive is useless.
Have any degaussing stories?
Like anyone else that gets a new toy we were excited to test this thing out. We had a few spare hard drives ready for destruction and wanted to see if degaussing even worked. The instructions were simple: put a drive in the tray, put the gloves on, and step on the peddle. Swirl the tray around in the provided circles for 10 seconds per side, and voila you'll have a cooked drive.
Sounded simple enough....
First we tried a 2.5" drive. BBBBBRRRRRRRRRRMMMMMMMMMMMMM...the degausser sounded off as it did its dirty work. Flip the drive over and BBBBBRRRRRRRRRRMMMMMMMMMMMMM...We pulled the drive off. Yep, it was toasty alright and it was accompanied by the the smell of hot metal. I took the drive back to my workstation after it had cooled off a bit. I wanted to see what had happened and wanted to see if the drive was still usable.
I plugged it in to my trusty writeblocker and turned on the writeblocker. The familiar whirring of the platters was quickly accompanied by a shocked "HOLY @$*%" coming out of my mouth. I could read the drive just fine. Not only was the drive intact, the filesystem was consistent and boy could I read the files. I quickly called over my teammates and we all looked at the drive contents as if we'd never seen a directory before.
Quickly I ran down the hall accompanied by another round of BBBBBRRRRRRRRRRMMMMMMMMMMMMM as the degausser chewed up another drive. This time it was a 3.5". I walked in to the room as the tray rattled around on top of the degausser. Not long after I entered the room was I walking back down the hall with two very hot hard drives in hand. I couldn't wait to see these not work.
I plugged in one drive and wouldn't you know it, I could read that one too. In disbelief I tried the other one. Yep I could read that one too. I pulled the drive from the write blocker and tried to format it. That actually did not work. Ok, so I could read the drive but could not write to it. That's reassuring but I care about someone being able to read it.
It was as if I'd been punched in the face. It was then that I recalled what the gentleman at EDR said when I was talking to him at technosecurity. He said "The problem with degaussers is that yeah sure, it's been marked as degaussed, but that's all you've got. Someone else saying they ran that drive through a degausser. What guarantee do you have that the drive is useless?"
Apparently I had no guarantee that the degausser works. If we couldn't get it to work properly, how would the people at the location actually housing this thing be able to ensure the drives were rendered useless. Promptly the vendor was contacted. They suggested we try to rotate the drive on another axis in addition to laying the drive flat. Willing to give it another try we turned the drive on its side and rotated it for the prescribed time. This time when I plugged in the drive, I could not read it. This was reassuring. Each successive time we tried a drive I was unable to read it.
This certainly reinforced the need to test all new equipment regardless of what its purpose is. It also reinforces the need to periodically verify the equipment to ensure its still working and you can trust it. Degaussing a drive is an interesting experience. I know that I for one would rather see a drive physically shredded, ground or smelted than degaussed. At least that way I can guarantee the drive is useless.
Have any degaussing stories?
sticking out
Commonly, when an intrusion occurs, an attacker will leave behind various types of detritus. Often times this is in the form of malware or a toolkit. In a recent compromise the attacker downloaded and left 80 MB of tools behind! This is admittedly not par for the course. Typically I'll see about 5-10 MB of detritus per incident. Identifying these files and classifying them is increasingly difficult. Consider the picture I've placed here. In a world where all items are of approximately the same color, this ball sticks out like an eye sore against the backdrop. How about in a filesystem where good files are known, an unknown file will stick out. Ah...if only that utopia existed. It can, if you invest in whitelisting a common build, but for most people out there that's just not reality. Unfortunately not only is identifying malware difficult, but what do you do with something you suspect of being malicious?Suppose you are a technician checking out a system that you suspect has been compromised. You check with your antivirus program and fail to detect anything strange. However, you notice something looks out of place. You could submit the malware to your antivirus vendor of choice, but depending on your licensing you may not get a response for 4-8 hours. Imagine malware living in your intranet for 4-8 hours because your vendor is slow. Believe me when I say that makes for a long day if you've never had to do it.
You could also bypass your vendor, get results quickly, develop a solution and roll it out in the time the vendor is working on a new definition. How would you do this?
As far as identification goes, what options are available? There are of course the more well known websites for submission. These sites have been mentioned and used by many for a long time now:
Virustotal
Anubis
CWSandbox
Norman
Jotti
Bit9
That's all well and good. I use these sites with some regularity, and the results have been good to date. There are some other options that you may or may not be aware of when you want to determine the badness(TM) of a file. Many of us are familiar with NSRL and other hashing projects and we should all be familiar with hashes and hashing files. Our tools do this for us automagically when we process an image, and if not, it's easy enough to generate a hashlist of the files in an image.
Suppose for a second that you have a list of files and their hashes, such as one generated by FTK Imager's directory listing capability. Suppose you want to take this list and look for possible malware. You could spend an obscene amount of money on a particular tool that matches hashes, or you could try some of these options:
Virus total hash matching
Offensivecomputing hash search
Team Cymru malware hash registry
And what about web based malware such as Javascript or Flash?
You could try these tools and sites:
Iseclabs Wepawet (The people behind Anubis)
Malzilla
For a little fun let's take a test drive of the Cymru malware hash registry.
First I'll hash some files(90% of which are malware):
loki:~ hogfly$ md5 *.exe
MD5 (724L1_setup_e.exe) = 848c95260d147543eff2e2c15acb58f1
MD5 (BR165652.exe) = d77d96af740af6805abcb2c572a758ce
MD5 (BR165680.exe) = 31c1d83de1db1aa5a806434d81183d79
MD5 (DNC-P-Ver.2.7.0.0.exe) = 518bcc3a6633dec8ceae3e0f02b4df60
MD5 (NATEON.exe) = af3c4884f690c48c115c8d9c55998141
MD5 (NVC.exe) = ee862735812241719960f2e069d99680
MD5 (abo.exe) = d2e23dbcbdd9a580b7897add524e4b09
MD5 (autorun.exe) = 3240c08878c7491b85b79c97db5c9204
MD5 (comrepl.exe) = 1d696a5dc70caa34d116344f50854d7f
MD5 (comrereg.exe) = 3619935460ddcb79f1ec9cc5710befc3
MD5 (dw8.exe) = f6da944f7c1ec3f0f8e6d673d9e9ff71
MD5 (envsetup.exe) = a1f8a82aad23a6b44cc92ee2eb1a10ef
MD5 (ff.exe) = 8a1b427981eecf67c60370b599c87dc6
MD5 (file.exe) = 107961dbceea53f729474b43c04302d4
MD5 (flvspeed.exe) = 45ce6d98337e4dba3e87d34adaf6d366
MD5 (hails.exe) = 5ebfe73e4fe237654a6bc07ed1712e7a
MD5 (hails2.exe) = 649d11e8d5676f0cee5c2a4a17f7e1a8
MD5 (index.exe) = 10980f4df2060b86a72eb5e533102980
MD5 (l07.exe) = 072ebc79aa1ff532c0d95f9a1ce4a395
MD5 (mfsl.exe) = 8fe25f71cbda9202995d74686eb5473e
MD5 (msdtc32.exe) = 205ca7ed3e6d8ae218c7fde2c50149f9
MD5 (net.exe) = c9c9a40e8a72907228e6a1bc9b5728ac
MD5 (net1.exe) = b8b857f3b5d8a8ef043fcf80120d0248
MD5 (omg.exe) = 654eef6ff6dbe666c1d9fd1f6049d525
MD5 (palzbn32.exe) = 28f02d257002221d367c0b43202c7a21
MD5 (pinyin.exe) = f9cbef1d67230b3845782b6fa11b976a
MD5 (rsscanner.exe) = a5953f3447a851f665702dd9afa63005
MD5 (scan.exe) = 29e20a4a5df73afee7acb3194f244b8e
MD5 (scann32.exe) = e464fb612104cc1da12c4d501cebe8df
MD5 (sss.exe) = 846790691b6f9717b9a1bf68e0bcd6e5
loki:~ hogfly$ whois -h hash.cymru.com 649d11e8d5676f0cee5c2a4a17f7e1a8
649d11e8d5676f0cee5c2a4a17f7e1a8 1224720131 31
loki:~ hogfly$ dig +short 649d11e8d5676f0cee5c2a4a17f7e1a8.malware.hash.cymru.com TXT
"1224720131 31"
loki:~ hogfly$ whois -h hash.cymru.com e464fb612104cc1da12c4d501cebe8df
e464fb612104cc1da12c4d501cebe8df 1221755478 25
So what do we have here? Two separate methods of doing a malware lookup (this is all explained on their site by the way). We simply feed them a hash and get back a unix datetime and the detection percentage of the malware. The detection percentage is a bit behind the curve and not all that accurate, but I would argue that it doesn't matter that much considering it's in their database and registered as malware. That doesn't mean I wouldn't like to see those numbers updated more frequently though. Cymru does support sending a large list of files via netcat for comparison, and I suspect that will be the most useful method of analysis for many out there.
For more active detection..If you've got about $25k I would strongly suggest you invest in a product called FireEye. I recently completed a demo of their product and all I can say is wow. I was most impressed. If you don't have the money for that you may want to try bothunter.
Here's to hoping you can identify and prosecute the investigation of suspicious files a little faster this year.
Subscribe to:
Posts (Atom)
