Friday, March 30, 2007

Honeynets with a twist

I've been running a honeynet for about 4 years now and in the past 6 months I've seen a dramatic shift in attacks. There are probably many people that would argue this, but the age of automatically spreading malware seems to be past. Before someone jumps down my throat let me explain. There will always be the vulnerability that is wormable, but with some of the protective measures being implemented by default now, a lot of this is mitigated. When was the last big global worm? There will always be the exploits that the kidiots download and run but seeing the same thing over and over again gets old fast when it's not a real situation.

For a time, I was fighting off FXP groups(which was very amusing actually), people loading up rootkits etc etc..but that seems to have dropped off. I think we're experiencing and have experienced a dramatic shift away from the automated kits that so plagued us for the past few years. The focus has become the user, and their gullability. There are some great honeynet tools out there now to go out and search websites but needless to say, my honeynet has gone dormant for the time being.

That said I've been considering modifying my honeynet to facilitate forensics and incident response, and not just for my own use. I haven't quite fleshed this idea out, but I'm wondering if it would even be considered useful by others.

We have hacking labs online for the public, why not forensics and incident response labs?
Imagine a network that was randomly infected with malware and compromises, and you had access to it, not only to check it out but actually test your response procedures and skills. Essentially what I'm saying is take The Hackers Challenge and make it a live situation. As I said I haven't fully fleshed this idea out, but would it be useful and interesting or just a waste of time?


Monday, March 26, 2007

Vista Complete Backups part 2

I spent a little more time with Windows backups today working on the shadow copy type of backup that's available. I started by looking at the Backup Rules registry key in HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsBackup\ScheduleParams\Rules
to decode the values in to something that actually makes sense. First a little explanation of the Key. Automatic backups can only be scheduled in Vista Ultimate and Business/Enterprise editions. So, if you're looking for this on a system that isn't one of these, good luck.
Under Rules you'll see a numerical value that corresponds to your drive letter. 1 is C, 2 is D and so on. These keys exist ONLY if you've selected them in your backup configuration. Let's look under key 1 shall we?
Here's what my system looks like. As you can see I am looking at my C drive backup configuration for automatic backups.

As I was looking at this I thought, what the heck is that hex value? It says flags, but what flags are they? Explanation forthcoming....

As you may have noticed, I like to use pictures to explain things (art is the international language right?)
Here's what I've gathered so far and I think it's fairly complete:
Each type of data in these backups is a power of 2 as shown here and please excuse my paint-fu or lack of.

As you can see I added in the numerical value of each data type that Windows Backup tries to use by default. Don't ask me why 16 is not there - if someone knows, please indulge me.

The backup flags begin counting at decimal value 1073741824 (0x40000000)
Add the data types in the image above to this number and you'll be able to decode any value you see.
Here's a quick cheat sheet:
Pictures only - 0x40000004 1073741828 - value of 4
Music - 0x40000001 1073741825 - value of 1
Videos - 0x40000002 1073741826 - value of 2
Documents - 0x40000008 1073741832 - value of 8
Email - 0x40000040 1073741888 - value of 64
TVshows - 0x40000080 1073741952 - value of 128
Compressed - 0x40000020 1073741856 - value of 32
Additional - 0x40000100 1073742080 - value of 256

All types selected - 0x400001ef 1073742319 - value of 495
It should be a matter of simple addition to decode the backup flags value now.

Pen testing and incidents

I was reading the latest information security mag and came across the face off column that Marcus Ranum and Bruce Schneier have. For those that haven't read this column before, the name of it "face off" tells you what it's all about.

It wasn't so much what they were debating that got me thinking, but rather it caused me to reflect back on a comment I once heard. I was sitting in a Hacking By Numbers session put on by SensePost. If you haven't had the opportunity to check them out, it's definitely worth it and the tools are great too. Anyways, it was during this session that we got in to a discussion about speed vs stealth of attacks on networks and systems. The old school of thought was that it was best to be stealthy while attacking networks. Making heavy use of IDS evasion techniques, clearing logs etc was the order of the day. After talking to these guys a little, then comparing what was said to the attacks I've seen and dealt with, the exact opposite is true. Speed seems to be the best option for many attackers, who cares if you splatter logs, the chances of being prosecuted for infiltrating a network is slim at best. If there was one thing worth remembering from the HBN session, it was something the instructor said...

"Stealth doesn't matter that much, I can set off your alarms 100 times or more as long as I accomplish my objective. I can screw up as many times as it takes to achieve my goal. You guys or your system administrators can only mess up once."

When I talk to sysadmins about incident preparation and detection I usually bring this up because the point is, we are up against a determined enemy and when it comes to protecting our networks and systems we need to be just as vigilant about protection as they are about accomplishing their objective.

Sunday, March 25, 2007

Damaged CD media

Last year I signed up for the DC3 forensics challenge while at the Blackhat conference. I never really got around to participating in the challenge due to time constraints though. I still have the packet sitting in my office and one day decided to play with the CD challenge. In this challenge I was presented with a CD that has some data on it, and it's been cut in to two unevenly sized pieces. The task is to retrieve the data. The data rings on the cd were small, roughly 50MB by the looks of it. Of course at this point, the 2006 challenge is over with and AccessData won the challenge but I never saw the reports posted anywhere. So, on to some CD fun!

I found the CD to be one of the more interesting challenges so I did some experimenting. In the name of experimentation some of the things I tried were:

Take a CD, write 50MB of data to it, cut it with an exacto knife, then trying to piece it back together using electrical tape. This actually worked...once. However it never worked in subsequent trials so this method was abandoned.

Take a CD, write 50MB of data to it, cut it with an exacto knife, then try to glue it to another CD. This failed miserably. The glue seeped in to the wrong places and the disc wouldn't budge in the reader.

Finally, I took a CD, wrote 50MB to it, and cut it with an exacto knife, then cut it again, and again and again and again, until I got something that resembled a credit card cd. This worked, over and over and over again. It worked in all 5 trials and was labeled the winner as you can see below what one disk looks like on the front and back.

As you might notice this particular test copy was broken - it's been in my bag for 6 months.

So, what's the trick here? Remember that CD's must maintain balance to spin properly inside of your reader. By cutting the disk to a smaller size, some extra weight was removed and the CD is no longer lopsided, so it balances easier and it now fits within the inner ring of the reader drive much like the credit card cd's do.

One other thing to remember, while experimenting with this some fun with a paper cutter was had. There's nothing like applying obscene amounts of cutting force to optical media!

The 2007 DC3 challenge has been posted on the DC3 site. Registration isn't open yet, but the topics look really interesting this year. Pay attention to the first three and you might notice that I've been talking a little about each one ;)
* BitLocker Breaker
* TPM Trickery
* Damaged Media Recovery
* Real vs. Fake Image Analysis
* SAM Attack
* Sizzling Thumb Drive
* Steganography Tracks
* Password Cracking
* Hack That
* and more...

A little rant

I've been waiting and waiting for a forensics equipment company to release evidence bags for digital evidence but I've yet to see it. I've searched, asked on mailing lists and still there isn't a single company that provides evidence bags for digital evidence. Seriously, digital evidence is mainstream now, and we should demand that suppliers provide us with evidence bags that allow us to provide an adquate description of the device contained within. In addition, when I bought my yearly supply last year not a single evidence bag was anti-static. I ended up buying anti-static bags from Uline and I stuff the evidence inside of them, seal the bag, then seal that bag inside of a generic evidence bag. So, vendors, if you ever read this blog or hear about it..provide us with proper equipment! The closest I've seen is from a company called SecurityBag

Other types of evidence have specific containers. Arson evidence is stored in metal cans, DNA swabs exist with caps to seal them, yet we can't properly identify our evidence on our bags.

This brings up another question I came up with recently, and maybe someone else has tried this already, but I wonder is the glue that holds together "tamper proof" bags succeptible to tampering? I think I'll have to sacrifice a few bags to testing.

Peer Review

One of the foundations of science is that scholarly work or ideas should go through a peer review process so the work can be validated and accepted or refuted and thrown out as an invalid theory. When it comes to digital forensic "science" I have to admit, I don't see much of this. Sure, we have a few scientific journals like Digital Investigation and organizations are popping up all over the place but I have yet to see any peer review being done of the core forensics processes. Take something as simple as disk imaging and ask two examiners how they acquire an image. I can almost guarantee you'll get two different answers. Does our field have any real requirement for image acquisition other than the hashes must match? Of course we must take precautions not to modify the contents of the evidence, but what if a suspect disk crashes while you're imaging it? Is that something we have any control over? Shouldn't you have been able to predict this event? We have tools like SMART to help predict disk failure, but how many of us collect this information before acquisition? Most of the time we can't even collect this information because our write blockers prevent the commands from reaching our disks!

How about how someone processes an image? What information is processed? How do you collect the information?

Forensics teams are popping up all over the place these days or the requirements are being built in to job descriptions of security folks, but how many of these groups actually get their processes peer reviewed? The answer is very very few. I know a few people that actually offer to peer review processes for others - for a price of course but this service isn't advertised.

This all points to one of the problems with our field. We don't yet have the scientific infrastructure in place to support promoting ourselves as scientists and our field as a science. One facet of the problem is that in many cases, digital forensics isn't considered a science, it's a business process and we all know that business processes are considered privileged information by the business community. Let me put it another way, companies don't disclose intrusion information because they fear public embarrassment or a drop in stock price. They also don't allow their people to disclose their forensics processes because they fear that something might be proven wrong or inadequate in the process.

Unfortunately, these processes can be dragged in to a court room at some point and be extremely damaging when the opposition rips apart what passes for an operational forensics process but not a legal one.

What I'm proposing is we develop a standard for the major activities carried out by a digital forensics examiner/team and a formal peer review for forensic processes. Maybe I've been going to the wrong conferences but to date, I've never seen anyone submit a formal process for public peer review (sorry but I don't consider $300 journals as a public forum -it's quite biased economically)

To put my money where my mouth is so to speak I'll soon be posting an acquisition procedure I wrote using Helix for review and validation as well as subsequent use/modification by anyone that is interested.

Friday, March 23, 2007

Vista Complete System Backup

Lately, I've been kicking around a few Vista Enterprise systems for testing and am working analyzing some of the "new features" of Vista. One such "new feature" is the Backup and Restore Center.
Microsoft decided to change the backup program a bit in this latest OS release. They finally moved away from the old interface they stole from Seagate and Veritas.

As you can see below, we now get the flashy GUI that tells us some interesting information. We can see when I made a complete disk image and when I tried out the shadow copy backup.

After making these backups I thought, hey wait..this has got to exist in the Registry. Like everything else in Windows there is indeed a Registry entry for the new Windows backup system.
You can check it out on your own systems here:

Underneath there I found a few subkeys, one of which is SystemImageBackup.
In here I found a series of values which I exported:

Windows Registry Editor Version 5.00


Interesting eh? We can see a few values that should be easily recognizable and one that at first threw me off a bit. The LastBackupTime and LastSuccessfulBackupTime values are a little large for 32bit hex values aren't they? I had to do a little searching on this one because it's listed as a REG_QWORD rather than REG_DWORD. Well, QWORD is a 64 bit value in little endian. As evidenced by Dcode You can correlate the time stamps between the image above and the decoded hex datetime stamp

Now, if you look back to the first image you'll see that the image was created on my E: drive. Obviously the registry values I'm showing keeps track of the Volume Information GUID as well as the logical drive letter. Now you have a volume GUID to match up when you start looking for missing disks from a system you are investigating.

After looking at this I decided to try to write a perl script using the Parse-Win32Registry module but have had some difficulty getting it to work. I keep getting this error while trying to use dumpreg.bat. If anyone knows of a fix to this or what the cause is, let me know please.

C:\Perl\site\bin>dumpreg.bat -r f:\vistareg\SOFTWARE Microsoft\Windows\CurrentVe
Invalid key node type at offset 0x38c8
38c8 a8 ff ff ff 6e 6b 20 10 3b 06 6d b8 fc 67 c7 01 ....nk .;.m..g..
38d8 00 00 00 00 20 00 00 00 b5 0c 00 00 00 00 00 00 .... ...........
38e8 b0 47 74 00 ff ff ff ff 00 00 00 00 ff ff ff ff .Gt.............
38f8 c0 ba 00 00 ff ff ff ff 8c 00 a0 00 00 00 00 00 ................
3908 00 00 00 00 00 00 00 00 6d 00 5c 00 07 00 00 00 ........m.\.....
at dumpreg.bat line 41

There are several other subkeys I'm looking at under this Key, including the ones created for automatic backup schedules.

Thursday, March 22, 2007

How do incidents occur?

As someone who thrives in what can only be described as organized chaos I get to answer the question "How did this happen?". Naturally when incidents occur, people like to blame the person lowest on the totem pole, which typically ends up being the system administrator. It is arguable whether or not anyone can blame the sysadmin that has to manage sometimes over 70 servers and 400 desktop computers but I tend to look at the organization holistically before I place blame on the lowely admins. Just as there is 'defense in depth' there are multiple layers of failure. Let's just say that failure(incidents), like security begins at the top of the organization. Take your highest ranking three letter exec and ask what their views on security are. Odds are you'll get a "secu-what?" response out of them most likely followed by a "I have people that work for me that do that sort of thing". Next, visit your financial officer and ask them what they think of security and what your security budget is this coming year. You might get a "well it's included in the IT budget" or "Good question, I'll get back to you on that". Then ask your C[TI]O/CISO what their take on security is. You'll probably end up getting a long winded response about the CIA triad of security and how your organization takes security seriously. Ask them how much time they allocate for their sysadmins to focus on security of their systems. Does it fall in that Other Duties as assigned category? Now go visit HR to ask what policies are in place regarding the security of your organization. There are lots of good policies in place in a lot of organizations but many places just say "yeah we have a policy about that". If you get that response, ask who is directly responsible for enforcing it, then ask that person about the policy. Then ask how they educate their people about the policy.

Travel down the exec path until you reach the system administrator who looks at you like your absolutely crazy as they show you this mornings help desk tickets and say "I don't have time to deal with security. I do what I can but no one gives me enough time to focus on security, so I fit it in when I can". Ask the admin what kind of access controls he or she has in place. You might be surprised one way or the other. Then ask them how they detect incidents. Ever seen a deer in headlights? You just did. Now ask them how they respond to incidents. Time for another one of those.."good question, let me find out" responses. At this point you proceed to slap your forehead and wonder..."how did it take so long for an incident to occur here?"

Does this sound at all familiar to anyone?

How do incidents occur? It's commonly a series of failures in the organization. It's sometimes amusing to pull all of the people listed above in to an incident response meeting. The poor guy at the top is fretting over how much money is being lost, the other execs are just dumbfounded because there are policies in place, and they take this security thing seriously. Meanwhile the sysadmin is sitting quietly at the end of the table wondering if their resume is up to date.

Let me recap. Incidents, like security begin at the top of the organizational chart. If the head of your organization isn't serious about it, then no one else will have the resources they need and incidents will occur, and more often than you might like. Just remember this little fact. In sports, when the team fails, it's not the players that usually get fired (occassionally they'll get traded yes), it's the coach. The coach is the one responsible for managing the efforts of the team below him or her.

Monday, March 19, 2007

Incident Response Teams

Over the weekend I went out and picked up a new game Ghost Recon for my computer (that's right I said my computer, not an Xbox). The objective, like many other first person shooter games where you lead teams is to save the day by completing your missions without getting your team killed.
In one scenario I was approaching an abandoned 18 wheeler trailer after sniping a bad guy by shooting his exposed knee caps and bang! my world came to a crashing halt. I'd been sniped and killed with a head shot. The game fades to black and white and you can hear your digital teammates calling your name as your vision blurs. Kind of cool..but it made me think of a few things.

I always push for the formation of incident response TEAMS when I can. In many cases a team will work much better than a single person. I recommend that at a minimum all response efforts involve two people. One person should handle the mainline incident response effort - investigation, live analysis etc. and the other person should handle backline response - the documentation, dealing with management, basically a person to take the administrative load of the incident off of the mainline responder(If you have a bigger team than this then of course it would vary). One of the core rules I learned early on was CYA (we all know what it means now). If I had had someone watching my back after I shot the bad guy in ghost recon, I probably wouldn't have been shot. When responding to incidents, try not to go it alone, or else you may end up as a casualty of the incident response world.

This brings me to my second point...
In the Ghost Recon scenario above..I didn't manage my team properly when I got shot. I had three other team members with me. One was a heavy weapons expert, another was an assault team member, and the last was a sniper. Had I been smart about this, I would have placed my sniper on overwatch somewhere, put the heavy weapons guy behind some cover with a good field of view, and had the assault team member move with me to the truck. Obviously I did none of that and wound up dead.

Incident Response Teams are many times groups of individuals with many specific skill sets, and it is the responsibility of the team leader to utilize the skillsets of their teams members to maximum efficiency. If you have a Windows Expert and a *nix expert on your team, it doesn't make much sense to send the Windows guy out as the lead incident handler when you have an incident on a Redhat box. Identify the areas of expertise in your teams before you send them out to respond and utilize their specialties when it makes sense to. Get your people trained, cross train them for redundancy but make sure you feed their expertise as well. i.e, all of my guys could shoot a gun, but they all had specialty weapons. Remember to never go out to an incident without the proper team and tools for the job.

When I load up ghost recon again, I'll remember to manage my team and utilize their skills and I bet you I won't get dead at the same spot ;)

Friday, March 16, 2007

Detecting BitLocker

Recently there's been a lot of concern about Bitlocker interfering with our ability to forensically analyze a system. I have to thank Microsoft for something...You've managed to force the issue of Live Response! Harlan Carvey has pointed out that Vista doesn't present any significant challenges to forensics, and I agree. But I do think it will modify the way we approach a system. In an effort to help people cope with their Bitlocker worries, I've written a REALLY BASIC script that will detect Bitlocker's presence on a computer. I decided to write it in VBS first with the hopes of porting it to perl very soon.

First off, let me say I'm no coder, I just dabble in a lot.

That said, Microsoft now provides two WMI classes that allow us to identify TPM usage and Bitlocker.
First we'll look at TPM. TPM is a hardware module that allows among other things, whole disk encryption. In Vista you can access and control TPM through the tpm console (tpm.msc). Once you've enabled and activated the TPM in the bios, you allow Vista to control what goes on. I'll go more in to this at a later time..

So, to access the TPM through WMI you need to use the new namespace:
and the class is Win32_Tpm.
More info is here

To take a look at bitlocker(which is far more interesting if you ask me) you need to use this namespace:
and the class is Win32_EncryptableVolume.
More on Bitlocker here

And now without further's my whopping Bitlocker detector!
strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\CIMV2\Security\MicrosoftVolumeEncryption")
Set colItems = objWMIService.ExecQuery( _
"SELECT * FROM Win32_EncryptableVolume",,48)
For Each objItem in colItems
Wscript.Echo "-----------------------------------"
Wscript.Echo "Bitlocker Encryptable Volumes"
Wscript.Echo "-----------------------------------"
Wscript.Echo "DeviceID: " & objItem.DeviceID
Wscript.Echo "DriveLetter: " & objItem.DriveLetter
Wscript.Echo "EncryptionMethod: " & objItem.GetEncryptionMethod

Here's what it looks like from my end:
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.

Bitlocker Encryptable Volumes
DeviceID: \\?\Volume{a1380f14-d3c6-11db-a066-00188b27a6b9}\
DriveLetter: F:
EncryptionMethod: 0

Obviously..I'm not using bitlocker yet, but my file system is ready for it.

Caveats: You need an administrative command prompt.

I'll keep plugging away on this..but it's a start.

Thursday, March 15, 2007

Thoughts on incident response

Richard Bejtlich was blogging about thoughts on incident response and asked that people respond in their own here goes..


The first 24 hours are crucial. Kind of a cliche but oh so true. If the wrong decision is made early, you can and will end up in a bad spot as the incident progresses.

Sacrifice Speed for GOOD DOCUMENTATION. Speed, while important should not replace your documentation. Remember..if it isn't written down, it didn't happen. Slow down and make sure your notes are up to snuff. I don't know how many times I've had to refer to my notes post incident..but without them I'd be lost.

The post incident follow up is just as important as the incident response effort. Always make sure you do this.
Some common questions:
What went right?
What went wrong?
What could have been done better?
Have all of the new security recommendations been implemented? If not, why?
Does the client require assistance with implementation?
What lessons have been learned?
How many man hours did the incident require?
Was any data lost during the response?
Was any hardware damaged?
Did the response procedures work?
Were the response procedures followed?
Did containment methods work?
Were there any tools that could have helped?
How soon after detection was the response initiated?
Was communication effective?
How long was the business operation of the department negatively impacted?
Was there adequate cooperation between the client Technical and administrative staff and the IRT?
Was the incident properly reported as per policy?

This phase also presents the IRT with the opportunity to review its own internal process and procedures for response efforts.

Items to review internally:
Were forensically sound procedures followed?
Was a lead handler assigned?
Were all communication channels used? Were the proper parties contacted?
Was the incident properly documented?
Do the procedures need to change?

Lessons Learned:

1) Obtain authority and determine the incident owner early. If you are not authorized to control a scene then what in the world are you doing there? Getting authorized to do XYZ will help get you past a lot of obstacles. I provide people with a form letter for just this purpose.

2) Prepare Emergency systems in advance. In one major incident, I ended up rebuilding a network infrastructure, including deployment of: switches, DHCP, NSM(debian running snort connected to a teeny tap), and had to rip apart two rather humorous looking switch closets. I now have images of a few types of systems ready and on hand.

3) You can either not sleep or not eat, but not both. On the same incident I was awake and on site for the first 48 hours, then got about 3 hours of sleep, because of the critical state of the incident. This incident took about 700 man hours to resolve. If you choose not to eat or sleep you'll run out of gas early in the incident and be unable to perform when you're needed most.

4) The war room is important. Make sure if your incident handling efforts take a while, you have adequate space on site that is a secure base of operations.

5) Get management involved. Until management starts to see the sometimes superhuman effort required to handle incidents, they will not understand the need to devote time and money to Incident Preparation.

Friday, March 2, 2007

Southeast Cybercrime Summit

I'll be headed to the CyberCrime Summit this year. I've never been able to make it but have heard great things about it. If anyone actually reads this blog and is headed there, maybe we can grab a drink.

Actus Reus and Corpus Delecti

In criminology terms like Actus Reus and corpus delecti are used to determine if a crime has been committed. Actus Reus is the latin term used for "guilty deed" or "deed of crime", while Corpus delecti is actual proof that a crime has been committed i.e, the dead body.

When it comes to Incidents though we can use it as a means to validate that an incident has actually occured. Have you ever been called in on a false alarm? Someone calls in a panic that their system must have been compromised but as it turns out it's a simple failure of a key service?

Have you ever stopped to think about how you reach your conclusion about an incident? That what you say happened, actually happened in the way you claim.

One of the things I'm actively working on behind the scenes are techniques to determine that not only did an incident occur but it happened in the way the investigator thinks. I intend to adapt the scientific method to the incident response and forensics processes many of us use. My hope is that this scientific approach will be a boon to investigations in that it will create a verifiable, repeatable process that provides predictive power in future investigations. Another hope is that it helps the investigator rule out the theories that many of us develop during the investigation leaving us with the explanation that makes the most sense because the other theories don't.