Sunday, March 25, 2007

Peer Review

One of the foundations of science is that scholarly work or ideas should go through a peer review process so the work can be validated and accepted or refuted and thrown out as an invalid theory. When it comes to digital forensic "science" I have to admit, I don't see much of this. Sure, we have a few scientific journals like Digital Investigation and organizations are popping up all over the place but I have yet to see any peer review being done of the core forensics processes. Take something as simple as disk imaging and ask two examiners how they acquire an image. I can almost guarantee you'll get two different answers. Does our field have any real requirement for image acquisition other than the hashes must match? Of course we must take precautions not to modify the contents of the evidence, but what if a suspect disk crashes while you're imaging it? Is that something we have any control over? Shouldn't you have been able to predict this event? We have tools like SMART to help predict disk failure, but how many of us collect this information before acquisition? Most of the time we can't even collect this information because our write blockers prevent the commands from reaching our disks!

How about how someone processes an image? What information is processed? How do you collect the information?

Forensics teams are popping up all over the place these days or the requirements are being built in to job descriptions of security folks, but how many of these groups actually get their processes peer reviewed? The answer is very very few. I know a few people that actually offer to peer review processes for others - for a price of course but this service isn't advertised.

This all points to one of the problems with our field. We don't yet have the scientific infrastructure in place to support promoting ourselves as scientists and our field as a science. One facet of the problem is that in many cases, digital forensics isn't considered a science, it's a business process and we all know that business processes are considered privileged information by the business community. Let me put it another way, companies don't disclose intrusion information because they fear public embarrassment or a drop in stock price. They also don't allow their people to disclose their forensics processes because they fear that something might be proven wrong or inadequate in the process.

Unfortunately, these processes can be dragged in to a court room at some point and be extremely damaging when the opposition rips apart what passes for an operational forensics process but not a legal one.

What I'm proposing is we develop a standard for the major activities carried out by a digital forensics examiner/team and a formal peer review for forensic processes. Maybe I've been going to the wrong conferences but to date, I've never seen anyone submit a formal process for public peer review (sorry but I don't consider $300 journals as a public forum -it's quite biased economically)

To put my money where my mouth is so to speak I'll soon be posting an acquisition procedure I wrote using Helix for review and validation as well as subsequent use/modification by anyone that is interested.


Mark McKinnon said...

Excellent post. I totally agree with everything that you are saying. Now I think part of the problem is that there are so many people out there that do not want to publish what they do so that the "other" side or "bad guys" does not get their hands on it. It goes along with having a checklist which is what the procedure is actually. Something else to think about to is that when you publish your procedure for acquisition I should be able to take it and with a little work and modification turn it into a procedure for the acquisition software I use since the principal should be the same as well as some of the steps (except software and OS).

hogfly said...

That's why I sometimes like being the little guy. I can actually focus on what matters (the science) and let some bureaucrat worry about "the bad guys" getting this information. It'd be nice to have the big guys budget, but that's another story.

What those who are unwilling to share fail to realize is that this information is out there already. The bad guys and the other side know the techniques and work daily to subvert detection which is precisely the reason I think it's rediculous to play this hand so close to our chests. I say share it now, before you get embarrased in court.

My hope in publishing a procedure is exactly what it sounds like you'd do with it. Take it, try it, comment on it, have a debate about it, modify it as you see fit, and then have something that at least two people can agree on. That's the whole idea behind peer review.

Great blog by the way. I've got a lot of reading to catch up on and your info is near the top (I got wind of it through cyberspeak)

Keydet89 said...


Great comments, and great post. I think what we're seeing now for IR/CF is what we saw in the late '90s with pen testing...everyone and his brother with a hacked ISS Internet Scanner license and a couple of freeware tools was a "pen tester", or an "ethical hacker" (oxymoron, I know).

I've been saying for a while that we don't have consistency in terminology...lawyers all know what "tort" means, and doctors all know what "stat" means. We lack specificity of language largely because what we do it misunderstood, or simply not understood.

I know what you mean about peer review. Even with a small team, sometimes I look at what someone else did and think to myself, "WTF??"

Aside from the journals, I'd like to see people write stuff that's actually usable, either on-scene or in the lab.

hogfly said...

You know it's actually quite frustrating that we don't have a taxonomy. I've read a few of them including the CERT doc for a common language for Computer Security Incidents and it's so old that it's useless now.