It was the head node in a high performance cluster with 64 nodes. With models and simulations being actively run on the system we naturally couldn't just power it down. So, we validate before escalation to investigation. The head node and subsequent systems were running linux and we just happened to have our handy cd containing trusted statically compiled binaries. Some of you might be saying..
The alert we received was nice enough to include a set of characteristics *cough* tool marks anyone?*cough* The tool marks listed were of the individual nature and even then, they varied. First things first. We need to capture memory. I liken memory captures to photographs of a crime scene, so we take our pictures before disturbing the system. Ok, so we grab a copy of /proc/kcore and kernel and symbol table and shoot them to the response laptop over a netcat connection. Then we attempted to locate the locations and files that were listed in the alert. Nothing, nothing and more nothing. Great news! But wait, just what happened here, we asked ourselves. Afterall we're responders and forensic analysts, we want to be able to understand and explain.
Tracing back through the alert, a username was identified and that's why we received the alert. The alerters thought "ahah we have a user name and an IP address that the user logged in to, hence the computer he logged in to is likely compromised". While we appreciated the alert, it was awfully presumptive. Yes the user in question logged in to the system we received the alert about from a computer in the foreign country where the alert originated - hence the user name and ip address of the system he logged in to. The system he logged in from, was identified as being compromised by those that sent us the alert, so they alerted us that one of our systems may have been compromised. This makes good sense but it still obviously required us to validate the compromise. Validation cost us a little effort but we certainly saved a bundle of time by not jumping to conclusions and going right in to investigation mode. The owners of the cluster would not have been happy if we had.