I had the pleasure of sitting down the other day with a technology law professor and we began discussing the issues surrounding memory capture. Not only the impact that a collection might have on a system and how a lawyer might attack that capture being presented as evidence, but the fact that the industry lacks standards and follows best practices, which presents its own set of issues.
I was pointed to a tort case involving tugboats of all things. The case of the TJ hooper is a rather important one it would seem. As this is an oft-cited case, I'll leave it to readers to google for it. In short, the case surrounded a cargo company, a transport company and a barge company. The transportation company assigned a number of tugboats to pull the barges along the atlantic coastline from Virginia to New Jersey. The cargo was lost in a storm and the cargo company claimed the barge company was at fault. The barge company in turn claimed it was the fault of the towing company. Not for any real fault of the tugboats - they were seaworthy vessels structurally. The judge however decided the tugs were unseaworthy because they did not have radios onboard.
Radios on tugboats in 1928...This was not the standard, or even common place in that era. However, some companies did have them. It was argued that radios were a best practice, and that the tug company should be faulted for not having them. The case ended up being about custom versus negligence. Was it customary for tugs to carry radios, and was the tug company negligent for not carrying them.
The professor I spoke with termed this "keeping up with the joneses" and said it was up to us to set the current best practices and ensure that these best practices are followed by all practitioners, lest they be held liable for negligence because others are performing live response and memory captures.
So, what is current best practice for live response?
Many would say that current best practice is to capture memory before doing anything else on the system, because it is the most volatile source of data.
I support this as a best practice, however there's a major problem with this idea; Judgement. The issue with following best practice rather than a standard is that judgement calls must be made. Make the wrong judgement call and you could be held liable for negligence. So, perhaps the industry needs to define cases for the application of best practice. Undoubtedly someone will respond to this and claim that these cases have already been defined. I would argue that sure they've been defined, but only in a cursory manner, and I would ask in what location can I find these best practices stated and which is actually the best practice when opinions differ?
Do we then begin to claim that one person is more credentialed than another and therefore the less credible persons opinions are not to be followed?
Other questions that some would ask:
When should you use a USB key instead of a network capture?
Should you capture memory twice during your response(before and after execution of response tools)?
EDIT 8/2/07: This bears clarification. It's been suggested that memory captures be taken twice during a response effort from two different devices. I've also been asked the above question before.
Is it best practice to capture memory before doing anything else on the system, even though the time it takes to capture memory can be prohibitive?
If you're in a race against time what do you do?
Do you need to run 40 tools in a script, when much of that information can be gathered from memory?
Do you need to run 40 tools in a script when 20 will suffice?
So, when you're performing a live response, remember the TJ hooper, and since we all follow best practice due to lack of standard, unless we all start dancing to the same tune, negligence will be hanging overhead at every turn.
EDIT: When I say I want cases, I don't literally mean legal cases, although it would be nice to have those as well. I was referring to cases as scenarios for when to and when not to do certain things.
Thursday, July 26, 2007
Friday, July 20, 2007
A did you know
Quite some time ago, Tableau stated that they'd be providing the ability to remove HPA and DCO's using their write blockers. Well, that day has come and gone. The Tableau Disk monitor tool provides this capability on windows. Cool. Now it seems according to Tim of sentinel-chicken that there is a linux equivalent to the HPA/DCO removal capability. VERY COOL! The tool is called tableau-parm and it's a command line interface. I'll be trying this tool out presently.
Tuesday, July 17, 2007
Modeling a scene
I always wanted to do something fancy for a presentation when it came to describing how I found a scene. Sure, you can use visio, or autocad if you want to get super fancy, or even some real expensive modeling software for crime scenes. I decided to experiment with modeling a simple sample scene today in google sketchup. It took me about 20 minutes, mainly because I'm new to the program in this capacity, however I really like what this program can do. It's easy to learn, has video tutorials, and models are abundant. Anyways, I exported my model to a 2d jpg(with varying degrees of detail)because there's no way to post a 3d model that I know of. The 3d model is much much better but oh well.
What do others use for modeling a scene?
What do others use for modeling a scene?
Monday, July 16, 2007
musings on bridging the gap and the blue wall
For a few years I've been trying to figure out how to pierce the blue wall to bridge the gap between law enforcement (government, state and local) such that people like myself can actually do some good with their skills. Why? Well when in school, after completing a project(details for those of you who are bored at the bottom), my CRJU prof said that LE could probably use a person like me. I found it inspiring at the time.
Some things I've learned along the way.
1) PD's don't respond to people
2) Pro-bono offerings don't get you anywhere
3) You must be "inned"
4) It takes a cop to work with them, or a professor
5) They don't want civilians seeing ugly all day long
There's a story here...I visited a state police cybercrime lab and spoke with the folks there. 60% of their cases were CP. I looked at the current pending cases board on the wall inside the exam room, and fully 90% of the cases at that time were CP. When asked what it's like, one examiner said "I see ugly all day long, I see it at work, I see it at home, I never stop seeing ugly, but I do what I can". Dealing with CP is tough work and this above all the other reasons is the one I can sympathize with.
For the most part, I'd be lying if I said I understood many of the reasons behind excluding the other half of the industry however I'm no fool, and no longer kid myself. The truth is, I'll never understand until I'm in the shoes of a LEO. Sure, I know several of them and I've heard stories, but that doesn't mean much, and it's not likely that I'd become an LEO.
As I've not yet been able to pierce the blue wall I offer an open invitation to anyone reading this blog that's a member of law enforcement of any kind. If there's anything that can be done(application analysis, testing, validation etc.) by a person such as myself, just shoot me a message. I have no hopes of contact, but like you all, I do what I can. If only LEO's would express their needs, the industry(commercial and individuals) I think could help. I've been all over the web, and I can't recall seeing an LEO sharing what it is that would help them the most. To that end I say keep on fighting the good fight, I'll continue to support the PBA, and just remember, there are people like me all over.
Thoughts, comments?
What was that project? Well, it was a mock search for a victim in an open area. We were given a scenario in one of my criminal investigation courses and asked to analyze the scene and determine a search pattern. I got a little crazy and went up in an airplane in January to take some aerials. Damn that was cold but so much fun. Some photos for your enjoyment(Thanks picasa!)..
One of many photos taken:
and my zone overlay.
Some things I've learned along the way.
1) PD's don't respond to people
2) Pro-bono offerings don't get you anywhere
3) You must be "inned"
4) It takes a cop to work with them, or a professor
5) They don't want civilians seeing ugly all day long
There's a story here...I visited a state police cybercrime lab and spoke with the folks there. 60% of their cases were CP. I looked at the current pending cases board on the wall inside the exam room, and fully 90% of the cases at that time were CP. When asked what it's like, one examiner said "I see ugly all day long, I see it at work, I see it at home, I never stop seeing ugly, but I do what I can". Dealing with CP is tough work and this above all the other reasons is the one I can sympathize with.
For the most part, I'd be lying if I said I understood many of the reasons behind excluding the other half of the industry however I'm no fool, and no longer kid myself. The truth is, I'll never understand until I'm in the shoes of a LEO. Sure, I know several of them and I've heard stories, but that doesn't mean much, and it's not likely that I'd become an LEO.
As I've not yet been able to pierce the blue wall I offer an open invitation to anyone reading this blog that's a member of law enforcement of any kind. If there's anything that can be done(application analysis, testing, validation etc.) by a person such as myself, just shoot me a message. I have no hopes of contact, but like you all, I do what I can. If only LEO's would express their needs, the industry(commercial and individuals) I think could help. I've been all over the web, and I can't recall seeing an LEO sharing what it is that would help them the most. To that end I say keep on fighting the good fight, I'll continue to support the PBA, and just remember, there are people like me all over.
Thoughts, comments?
What was that project? Well, it was a mock search for a victim in an open area. We were given a scenario in one of my criminal investigation courses and asked to analyze the scene and determine a search pattern. I got a little crazy and went up in an airplane in January to take some aerials. Damn that was cold but so much fun. Some photos for your enjoyment(Thanks picasa!)..
One of many photos taken:
and my zone overlay.
Saturday, July 14, 2007
The rise of network forensics
I am starting to think that "network forensics" is going to quickly become the next "big thing"(TM) in the digital forensics discipline.
Well, what is network forensics?
by definition:Network forensics is the capture, recording, and analysis of network events in order to discover the source of security attacks or other problem incidents.
Unfortunately, that's the only definition I could actually find. Why is that I wonder? Perhaps because there is no such thing as network forensics? Is it just really another name for network security monitoring or intrusion analysis?
NSM is the collection, analysis and escalation of indications and warnings to detect and respond to intrusions. Intrusion analysis is much the same.
Let's look at a few of the tools used for each shall we?
Tcpdump
Wireshark
ngrep
tcpreplay
P0f
net flows(cisco, argus etc)
Snort or other IDS
Qradar
These are obviously only a few tools, many open source. Either way, they are multi-purpose tools used for things like protocol analysis, traffic analysis, intrusion analysis, security monitoring and now...network forensics? Let's be honest with ourselves, network "forensics" appears to be just a buzzword for what's been done for years in the analyst field. There's nothing "forensic" about it..or is there?
So, where does the "forensic" component come in to play? Simple. Collection, preservation, and presentation as evidence.
Network forensics has many criteria that must be met in order for it to be useful as evidence.
My initial thoughts...
1) It must be complete
Whether it's session data, full content, alert or other, it must be complete. A capture that's missing packets could be seen as damaging to a case. A capture system must be capable of keeping up with the network it's connected to. Anomaly systems like Qradar that decide what to capture content on, or not, should not be used as evidence because the capture will be incomplete. A partial capture could be used to perhaps get a wiretap, or warrant but presenting it as fact or of evidentiary value would jeopardize a case in my opinion.
2) It must be authentic
Captures must be proven to be authentic. Hashing routines should be run as a capture log rolls over or each packet should be hashed as it's stored, before the analysis begins. In addition, the capture should be done by two independent systems if possible.
3) Time is critical
The collection system must be synched, and any time disparity must be accounted for.
4) Analysis systems need logging capabilities.
An analyst looking at a network capture to be used as evidence must allow for logging of the actions of the analyst.
5) Anyone presenting network based evidence must be trained in packet and protocol analysis, not simply the tools used.
In my opinion, being able to read a packets content in depth and being able to accurately interpret and analyze it is of utmost importance. Being able to explain the why's and how's is critical. It's easy to jump to conclusions that are incorrect with network based evidence. Richard rebuffs here. And here is an incorrect conclusion. It's very easy to do since many scenarios will fit a similar signature.
6) It must be used in conjuction with host based forensics if possible.
Of course, not every scenario is ideal, but remember that the power of digital evidence is in the ability to corroborate. Corporal and Environmental evidence should be used to corroborate each other if an accurate reconstruction is to take place. The value of the evidence will be bolstered if the two sources support each other.
7) Sensor deployment must be appropriate.
It does almost no good to deploy a sensor at the perimeter if the attack is internal. It might make sense to deploy multiple sensors in this type of case. Care must be taken to deploy at the right time and location.
What are others thoughts on the subject of network forensics? Is it snake oil, or a bonafide digital forensics specialty?
Well, what is network forensics?
by definition:Network forensics is the capture, recording, and analysis of network events in order to discover the source of security attacks or other problem incidents.
Unfortunately, that's the only definition I could actually find. Why is that I wonder? Perhaps because there is no such thing as network forensics? Is it just really another name for network security monitoring or intrusion analysis?
NSM is the collection, analysis and escalation of indications and warnings to detect and respond to intrusions. Intrusion analysis is much the same.
Let's look at a few of the tools used for each shall we?
Tcpdump
Wireshark
ngrep
tcpreplay
P0f
net flows(cisco, argus etc)
Snort or other IDS
Qradar
These are obviously only a few tools, many open source. Either way, they are multi-purpose tools used for things like protocol analysis, traffic analysis, intrusion analysis, security monitoring and now...network forensics? Let's be honest with ourselves, network "forensics" appears to be just a buzzword for what's been done for years in the analyst field. There's nothing "forensic" about it..or is there?
So, where does the "forensic" component come in to play? Simple. Collection, preservation, and presentation as evidence.
Network forensics has many criteria that must be met in order for it to be useful as evidence.
My initial thoughts...
1) It must be complete
Whether it's session data, full content, alert or other, it must be complete. A capture that's missing packets could be seen as damaging to a case. A capture system must be capable of keeping up with the network it's connected to. Anomaly systems like Qradar that decide what to capture content on, or not, should not be used as evidence because the capture will be incomplete. A partial capture could be used to perhaps get a wiretap, or warrant but presenting it as fact or of evidentiary value would jeopardize a case in my opinion.
2) It must be authentic
Captures must be proven to be authentic. Hashing routines should be run as a capture log rolls over or each packet should be hashed as it's stored, before the analysis begins. In addition, the capture should be done by two independent systems if possible.
3) Time is critical
The collection system must be synched, and any time disparity must be accounted for.
4) Analysis systems need logging capabilities.
An analyst looking at a network capture to be used as evidence must allow for logging of the actions of the analyst.
5) Anyone presenting network based evidence must be trained in packet and protocol analysis, not simply the tools used.
In my opinion, being able to read a packets content in depth and being able to accurately interpret and analyze it is of utmost importance. Being able to explain the why's and how's is critical. It's easy to jump to conclusions that are incorrect with network based evidence. Richard rebuffs here. And here is an incorrect conclusion. It's very easy to do since many scenarios will fit a similar signature.
6) It must be used in conjuction with host based forensics if possible.
Of course, not every scenario is ideal, but remember that the power of digital evidence is in the ability to corroborate. Corporal and Environmental evidence should be used to corroborate each other if an accurate reconstruction is to take place. The value of the evidence will be bolstered if the two sources support each other.
7) Sensor deployment must be appropriate.
It does almost no good to deploy a sensor at the perimeter if the attack is internal. It might make sense to deploy multiple sensors in this type of case. Care must be taken to deploy at the right time and location.
What are others thoughts on the subject of network forensics? Is it snake oil, or a bonafide digital forensics specialty?
Thursday, July 12, 2007
Determining memory consumption
As I've been developing the methodology I talked about previously , one of the problem areas that's arisen is determining system impact. One component of impact is determining memory consumption. There's a lot of work to be done here because of the complexity of memory management, and there's a lot of work being done currently. If we're after precision, we need to know how many pages have been allocated to a process, how many are used, if they are in swap or resident in memory, what was previously in those pages, and what's in them now.
I started working on a primitive method to determine memory consumption. Note I said consumption rather than displacement, since determining displacement requires a lot of wrench time.
One such method to maybe start determining consumption is to use the debugging tools from Microsoft. The two tools I'll use here are windbg and cdb(same tool actually..just one is command line).
The tool I'm checking out, is another tool included in the debugging toolkit, also used in incident response and live response scenarios.
So..here goes.
fire up a command prompt and run this:
C:\Program Files\Debugging Tools for Windows>cdb.exe -y http://msdl.microsoft.com/download/symbols -o tlist.exe -t
Microsoft (R) Windows Debugger Version 6.7.0005.1
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: tlist -t
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 01000000 012dd000 tlist.exe
ModLoad: 7c900000 7c9b0000 ntdll.dll
ModLoad: 7c800000 7c8f4000 C:\WINDOWS\system32\kernel32.dll
ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll
ModLoad: 03000000 03116000 C:\Program Files\Debugging Tools for Windows\dbghel
p.dll
ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
ModLoad: 77e70000 77f01000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77c00000 77c08000 C:\WINDOWS\system32\VERSION.dll
ModLoad: 7e410000 7e4a0000 C:\WINDOWS\system32\USER32.dll
ModLoad: 77f10000 77f57000 C:\WINDOWS\system32\GDI32.dll
ModLoad: 774e0000 7761d000 C:\WINDOWS\system32\ole32.dll
ModLoad: 77120000 771ac000 C:\WINDOWS\system32\OLEAUT32.dll
(7c.f90): Break instruction exception - code 80000003 (first chance)
eax=00181eb4 ebx=7ffd5000 ecx=00000001 edx=00000002 esi=00181f48 edi=00181eb4
eip=7c901230 esp=0006fb20 ebp=0006fc94 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
ntdll!DbgBreakPoint:
7c901230 cc int 3
0:000>
Ok so that's good. Now we're stepping in to the tlist process and we've reached the first breakpoint. Now, fire up windbg and attach to the kernel (CTRL+K), select local and start with !process 0 0 to get a list of processes that are active.
lkd> !process 0 0
PROCESS 82b7c020 SessionId: 0 Cid: 007c Peb: 7ffd5000 ParentCid: 0da8
DirBase: 03660360 ObjectTable: e12f6a48 HandleCount: 7.
Image: tlist.exe
Aha, here's tlist. Now, we can key in on the process (output truncated):
lkd> !process 82b7c020
PROCESS 82b7c020 SessionId: 0 Cid: 007c Peb: 7ffd5000 ParentCid: 0da8
DirBase: 03660360 ObjectTable: e12f6a48 HandleCount: 7.
Image: tlist.exe
VadRoot 829ff7f8 Vads 27 Clone 0 Private 46. Modified 0. Locked 0.
DeviceMap e1b2efd0
Token e1165d48
ElapsedTime 00:05:54.546
UserTime 00:00:00.015
KernelTime 00:00:00.062
QuotaPoolUsage[PagedPool] 27892
QuotaPoolUsage[NonPagedPool] 1080
Working Set Sizes (now,min,max) (236, 50, 345) (944KB, 200KB, 1380KB)
PeakWorkingSetSize 236
VirtualSize 12 Mb
PeakVirtualSize 12 Mb
PageFaultCount 228
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 829
DebugPort 82e50a60
Look at the bolded text. We can see the process working set size. At present, the tlist process has been allocated 236 pages of memory at 4k. Multiply 236 * 4 and you'll get 944K. So, at the initial BP we see that tlist is "using" 944K of memory.
In the CDB window, if you tell it to 'go' by typing 'g' we'll see what happens to memory usage.
0:000> g
System Process (0)
System (4)
smss.exe (604)
csrss.exe (652)
winlogon.exe (676)
services.exe (720)
svchost.exe (904)
svchost.exe (1004)
svchost.exe (1116)
wuauclt.exe (3272)
svchost.exe (1232)
svchost.exe (1412)
ccSetMgr.exe (1608)
ccEvtMgr.exe (1656)
SPBBCSvc.exe (1772)
spoolsv.exe (428)
DefWatch.exe (332)
Rtvscan.exe (1100) Scan
VMwareService.exe (1384)
alg.exe (1976)
svchost.exe (1916)
wmiapsrv.exe (1444)
lsass.exe (740)
explorer.exe (1528) Program Manager
VMwareTray.exe (632)
VMwareUser.exe (940)
ccApp.exe (944)
VPTray.exe (816) Missing Virus Definitions
PicasaMediaDetector.exe (936) Picasa Media Detector
taskmgr.exe (3252) Windows Task Manager
cmd.exe (3904) Command Prompt - cdb.exe -y http://msdl.microsoft.com/download
/symbols -o tlist -t
windbg.exe (3500) Windows Driver Kit: Debugging Tools
cdb.exe (3496)
tlist.exe (124)
mmc.exe (3216) Performance
hh.exe (4092) Windows Driver Kit: Debugging Tools
eax=0300b7f8 ebx=00000000 ecx=002643e8 edx=00260608 esi=7c90e88e edi=00000000
eip=7c90eb94 esp=0006fe44 ebp=0006ff40 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c90eb94 c3 ret
Groovy, now we get our tlist output and the tool is done running. However, since we haven't freed the process from the debugger, we have a thread in wait state, which means we can figure out how much memory our process actually consumed:
key in on the same process as before in windbg and you'll see the following:
lkd> !process 82b7c020
PROCESS 82b7c020 SessionId: 0 Cid: 007c Peb: 7ffd5000 ParentCid: 0da8
DirBase: 03660360 ObjectTable: e12f6a48 HandleCount: 22.
Image: tlist.exe
VadRoot 82bc3840 Vads 36 Clone 0 Private 380. Modified 0. Locked 0.
DeviceMap e1b2efd0
Token e1165d48
ElapsedTime 00:12:59.265
UserTime 00:00:00.015
KernelTime 00:00:00.140
QuotaPoolUsage[PagedPool] 38236
QuotaPoolUsage[NonPagedPool] 1440
Working Set Sizes (now,min,max) (771, 50, 345) (3084KB, 200KB, 1380KB)
PeakWorkingSetSize 771
VirtualSize 18 Mb
PeakVirtualSize 19 Mb
PageFaultCount 765
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 1120
DebugPort 82e50a60
So, we see 3084K has been "used", but can that possibly be accurate? The answer of course is no. There are several factors at play.
1) This was run from a debugger - this adds memory allocated to the process to account for debugging.
2) There is shared memory being used by other DLL's.
3) The working set is not representative of memory actually used by the process. It's representative of the memory(shared, virtual, physical) allocated to the process, but not necessarily that which is consumed(or resident). In addition, Microsoft's documentation on MSDN is inconsistent(surprise, surprise) in describing what's actually included in the working set.
4) The working set is what is recorded by the task manager, which is inaccurate.
So, where does this leave us? Well, in terms of precision and accuracy for determining memory consumption for a process, the working set is not the answer because the allocation hasn't necessarily been consumed, maybe the private bytes associated with a process is what we need to focus on but the real question is, and this has yet to be answered..what's good enough? I'm thinking that maybe the working set size is good enough, but not necessarily precise.
Next steps? Boot a system in debug mode and attach a kernel debugger to it while it's live.
Thoughts?
EDITS: the url is download, not downloads.
I started working on a primitive method to determine memory consumption. Note I said consumption rather than displacement, since determining displacement requires a lot of wrench time.
One such method to maybe start determining consumption is to use the debugging tools from Microsoft. The two tools I'll use here are windbg and cdb(same tool actually..just one is command line).
The tool I'm checking out, is another tool included in the debugging toolkit, also used in incident response and live response scenarios.
So..here goes.
fire up a command prompt and run this:
C:\Program Files\Debugging Tools for Windows>cdb.exe -y http://msdl.microsoft.com/download/symbols -o tlist.exe -t
Microsoft (R) Windows Debugger Version 6.7.0005.1
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: tlist -t
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 01000000 012dd000 tlist.exe
ModLoad: 7c900000 7c9b0000 ntdll.dll
ModLoad: 7c800000 7c8f4000 C:\WINDOWS\system32\kernel32.dll
ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll
ModLoad: 03000000 03116000 C:\Program Files\Debugging Tools for Windows\dbghel
p.dll
ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
ModLoad: 77e70000 77f01000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77c00000 77c08000 C:\WINDOWS\system32\VERSION.dll
ModLoad: 7e410000 7e4a0000 C:\WINDOWS\system32\USER32.dll
ModLoad: 77f10000 77f57000 C:\WINDOWS\system32\GDI32.dll
ModLoad: 774e0000 7761d000 C:\WINDOWS\system32\ole32.dll
ModLoad: 77120000 771ac000 C:\WINDOWS\system32\OLEAUT32.dll
(7c.f90): Break instruction exception - code 80000003 (first chance)
eax=00181eb4 ebx=7ffd5000 ecx=00000001 edx=00000002 esi=00181f48 edi=00181eb4
eip=7c901230 esp=0006fb20 ebp=0006fc94 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
ntdll!DbgBreakPoint:
7c901230 cc int 3
0:000>
Ok so that's good. Now we're stepping in to the tlist process and we've reached the first breakpoint. Now, fire up windbg and attach to the kernel (CTRL+K), select local and start with !process 0 0 to get a list of processes that are active.
lkd> !process 0 0
PROCESS 82b7c020 SessionId: 0 Cid: 007c Peb: 7ffd5000 ParentCid: 0da8
DirBase: 03660360 ObjectTable: e12f6a48 HandleCount: 7.
Image: tlist.exe
Aha, here's tlist. Now, we can key in on the process (output truncated):
lkd> !process 82b7c020
PROCESS 82b7c020 SessionId: 0 Cid: 007c Peb: 7ffd5000 ParentCid: 0da8
DirBase: 03660360 ObjectTable: e12f6a48 HandleCount: 7.
Image: tlist.exe
VadRoot 829ff7f8 Vads 27 Clone 0 Private 46. Modified 0. Locked 0.
DeviceMap e1b2efd0
Token e1165d48
ElapsedTime 00:05:54.546
UserTime 00:00:00.015
KernelTime 00:00:00.062
QuotaPoolUsage[PagedPool] 27892
QuotaPoolUsage[NonPagedPool] 1080
Working Set Sizes (now,min,max) (236, 50, 345) (944KB, 200KB, 1380KB)
PeakWorkingSetSize 236
VirtualSize 12 Mb
PeakVirtualSize 12 Mb
PageFaultCount 228
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 829
DebugPort 82e50a60
Look at the bolded text. We can see the process working set size. At present, the tlist process has been allocated 236 pages of memory at 4k. Multiply 236 * 4 and you'll get 944K. So, at the initial BP we see that tlist is "using" 944K of memory.
In the CDB window, if you tell it to 'go' by typing 'g' we'll see what happens to memory usage.
0:000> g
System Process (0)
System (4)
smss.exe (604)
csrss.exe (652)
winlogon.exe (676)
services.exe (720)
svchost.exe (904)
svchost.exe (1004)
svchost.exe (1116)
wuauclt.exe (3272)
svchost.exe (1232)
svchost.exe (1412)
ccSetMgr.exe (1608)
ccEvtMgr.exe (1656)
SPBBCSvc.exe (1772)
spoolsv.exe (428)
DefWatch.exe (332)
Rtvscan.exe (1100) Scan
VMwareService.exe (1384)
alg.exe (1976)
svchost.exe (1916)
wmiapsrv.exe (1444)
lsass.exe (740)
explorer.exe (1528) Program Manager
VMwareTray.exe (632)
VMwareUser.exe (940)
ccApp.exe (944)
VPTray.exe (816) Missing Virus Definitions
PicasaMediaDetector.exe (936) Picasa Media Detector
taskmgr.exe (3252) Windows Task Manager
cmd.exe (3904) Command Prompt - cdb.exe -y http://msdl.microsoft.com/download
/symbols -o tlist -t
windbg.exe (3500) Windows Driver Kit: Debugging Tools
cdb.exe (3496)
tlist.exe (124)
mmc.exe (3216) Performance
hh.exe (4092) Windows Driver Kit: Debugging Tools
eax=0300b7f8 ebx=00000000 ecx=002643e8 edx=00260608 esi=7c90e88e edi=00000000
eip=7c90eb94 esp=0006fe44 ebp=0006ff40 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c90eb94 c3 ret
Groovy, now we get our tlist output and the tool is done running. However, since we haven't freed the process from the debugger, we have a thread in wait state, which means we can figure out how much memory our process actually consumed:
key in on the same process as before in windbg and you'll see the following:
lkd> !process 82b7c020
PROCESS 82b7c020 SessionId: 0 Cid: 007c Peb: 7ffd5000 ParentCid: 0da8
DirBase: 03660360 ObjectTable: e12f6a48 HandleCount: 22.
Image: tlist.exe
VadRoot 82bc3840 Vads 36 Clone 0 Private 380. Modified 0. Locked 0.
DeviceMap e1b2efd0
Token e1165d48
ElapsedTime 00:12:59.265
UserTime 00:00:00.015
KernelTime 00:00:00.140
QuotaPoolUsage[PagedPool] 38236
QuotaPoolUsage[NonPagedPool] 1440
Working Set Sizes (now,min,max) (771, 50, 345) (3084KB, 200KB, 1380KB)
PeakWorkingSetSize 771
VirtualSize 18 Mb
PeakVirtualSize 19 Mb
PageFaultCount 765
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 1120
DebugPort 82e50a60
So, we see 3084K has been "used", but can that possibly be accurate? The answer of course is no. There are several factors at play.
1) This was run from a debugger - this adds memory allocated to the process to account for debugging.
2) There is shared memory being used by other DLL's.
3) The working set is not representative of memory actually used by the process. It's representative of the memory(shared, virtual, physical) allocated to the process, but not necessarily that which is consumed(or resident). In addition, Microsoft's documentation on MSDN is inconsistent(surprise, surprise) in describing what's actually included in the working set.
4) The working set is what is recorded by the task manager, which is inaccurate.
So, where does this leave us? Well, in terms of precision and accuracy for determining memory consumption for a process, the working set is not the answer because the allocation hasn't necessarily been consumed, maybe the private bytes associated with a process is what we need to focus on but the real question is, and this has yet to be answered..what's good enough? I'm thinking that maybe the working set size is good enough, but not necessarily precise.
Next steps? Boot a system in debug mode and attach a kernel debugger to it while it's live.
Thoughts?
EDITS: the url is download, not downloads.
Tuesday, July 10, 2007
New ACPO guidelines
The ACPO released their new guidelines recently. I really have to hand it to the ACPO for reviewing their own guidelines on a regular basis and keeping up with new techniques and technology.
Of particular interest in the document is the Network Forensics and Volatile data section.
"By profiling the forensic footprint of
trusted volatile data forensic tools, an investigator will be
in a position to understand the impact of using such tools
and will therefore consider this during the investigation
and when presenting evidence."
It's about time that this statement was made in an official manner. While I'm in the process of actually defining a methodology to run these tests, it's really nice to see this.
It was also important for the ACPO to include the note about the Trojan Defense.
"Considering a potential Trojan defence, investigators
should consider collecting volatile evidence. Very often,
this volatile data can be used to help an investigator
support or refute the presence of an active backdoor."
Great inclusion!
The ACPO seems to hint at the requirement to add network forensics procedures due to trojan defense claims and the apparently large amount of claims that "the trojan did it".
Any of you fine folks across the pond have a metric for the number of claims?
It would also seem as if Network Forensics will be a major focal point in the upcoming months for many investigators.
I have to commend the ACPO for releasing these guidelines, it's a great resource.
Of particular interest in the document is the Network Forensics and Volatile data section.
"By profiling the forensic footprint of
trusted volatile data forensic tools, an investigator will be
in a position to understand the impact of using such tools
and will therefore consider this during the investigation
and when presenting evidence."
It's about time that this statement was made in an official manner. While I'm in the process of actually defining a methodology to run these tests, it's really nice to see this.
It was also important for the ACPO to include the note about the Trojan Defense.
"Considering a potential Trojan defence, investigators
should consider collecting volatile evidence. Very often,
this volatile data can be used to help an investigator
support or refute the presence of an active backdoor."
Great inclusion!
The ACPO seems to hint at the requirement to add network forensics procedures due to trojan defense claims and the apparently large amount of claims that "the trojan did it".
Any of you fine folks across the pond have a metric for the number of claims?
It would also seem as if Network Forensics will be a major focal point in the upcoming months for many investigators.
I have to commend the ACPO for releasing these guidelines, it's a great resource.
Subscribe to:
Posts (Atom)
