Live Response Tool Testing

Over on the Windows Forensic Analysis group there was a recent discussion about live response tool invasiveness and some methods that people currently use. I started thinking about how to codify the methods used so that a standard methodology is created that can be used by anyone.

Below is a copy of my message to that list. I hope readers of this blog can respond with suggestions and any comments.
--

In order to effectively begin to measure the effects our tools have on systems we need to devise a testing methodology. Harlan recently asked "what's meant by testing of tools"?

I'd like to take a first whack at this one to see what others think. I'm hoping people add something to this. One thing to understand is that live response by its nature is not repeatable. However, testing in a controlled environment is as long as variables are identified and documented. Using a standard methodology will allow us to create a system by which our efforts can be measured, and our actions will be much more defensible as a result.

Goals:
To create a standard methodology for measuring the effects of live response tools on Windows Operating Systems.

Constraints and scope:
1) This methodology will be for live response tools only.

2) The methodology will measure the effects of the tool on: physical & virtual memory, the file system, registry, and network state. Modification will be measured in two categories. Type of data being modified and a quantifiable amount of each type of data being modified(i.e, a list of file modifications, amount of memory displaced etc).

3) The control system must be isolated and an accurate baseline must be established.

Needs:
Define an accurate baseline.
Identify tools and current methodogies for using each.
Create a standard baseline image.
Define the process for measuring the effects of the tool.