Responder Pro - A review

Here's a short disclaimer before I get in to this.
*I'm not paid by nor affiliated with HBGary. This is an honest review of their product(s).*

A short while ago I received a demo copy of HBGary's Responder Pro product. A big thanks goes out to Rich and the HBGary team for letting me demo their tools. My demo period has now expired so I wanted to share my experience.

During my demo I used Responder Pro almost exclusively to analyze malware, and perform memory analysis. There's a bit of a learning curve with the product, mainly in getting used to the layout of the GUI which was at first a senseless morass of windows and tabs. After I adapted my thinking and used the tool a few times, the GUI made some sense.

Once I got acclimated to the GUI, memory analysis couldn't have been any easier. The GUI is pretty powerful and allows for a quick examination of the 'big win' components of memory - processes, modules, open files, open registry keys, network connections. Identifying process and DLL injection was in a word 'simple' once I figured out how the tool laid out the process and module information. Image(executable) extraction is simple - a right click does the trick.

A warning though. If you're using Antivirus products on the system you use this tool on, be prepared to redo your analysis or make exceptions for files and folders. More than once I was frustrated by having Symantec Endpoint Protection delete the extracted binary, leaving Responder in a state of confusion and inability to complete an analysis. I have many v.2 case files due to this.

The automated malware analysis of the memory dump was a huge timesaver. Based on a file called baserules.txt, a memory dump will be analyzed for processes and modules that are exhibiting potentially malicious behaviors. If you highlight a module, it will be selected for a deeper dive analysis. Did I mention it's a time saver? Analyzing module after module in a process can be tedious work. Having the information presented to you allows you to quickly weed out what looks normal from the abnormal.

My one nit about the automated analysis was the transition from 1.3 to 1.4. 1.4 had far too many rules commented out, and while this led to fewer false positives, it greatly contributed to more manual work because it missed a lot of things.

During my demo period HBGary updated Responder Pro from version 1.3 to version 1.4. The transition added interesting capabilities such as pulling out URL's from the memory dump as well as passwords. Harlan discussed this a bit while looking at one of my memory snapshot project images.

Memory analysis-wise Responder is right up there for commercial tools. I'd pretty much say it's the best around for the price point ($1000 for Field edition). It also integrates with Encase, which is nice for a lot of people.

And then there's the graphing for malware analysis. One of my colleagues summed it up accurately by calling it very 'seductive'. Now, graphing has been around a while for malware analysis. There's a difference though when it comes to using Responder. The difference is you don't have to screw around with the reindeer games that various packers use. When you're analyzing a memory dump of malware, you're seeing the unpacked malware and it makes for a very straightforward analysis. In more than one case I was able to do analysis in about an hour or so on something that would have otherwise taken a few hours. The ability to pull out a subroutine, and analyze it graphically and having the code available as well is a fantastic feature. Or, if you want to, you can begin by performing an analysis of a process, and looking at the strings. Then just pull the string you're interested in, in to the working canvas, and begin analysis on something that looks like it's of direct interest to you. That's what I was doing here. The bookmarking and layering made it almost photshop'esque. I only had to look at what was of interest and I could go back to it later. While analyzing virut.CF the bookmarking feature was very handy, especially when I discovered some Passthru driver configuration files intact while doing a graphical analysis. I won't get in to the differences between IDA pro and Responder Pro for analysis but I will say that I had a much faster time of doing analysis in Responder than in IDA, and I think the reason was due to using a memory dump rather than static binary analysis.

So that's enough talking about why I like the product. Case Study-wise I used Responder Pro to look at several poorly classified malware types during my demo. In the field I use Responder Pro to analyze several USB related malware variants that my other vendors called "downloader" or "trojan horse" or "SillyFDC". In a wave of compromises I didn't want any other tool for analysis. I reached for Responder Pro when I needed to do an analysis to determine scope and the REAL risk to data. I reached for Responder Pro when I needed to determine the capabilities of a few very nasty pieces of malware. Why? Because I needed accurate, actionable intel fast.

Just this evening I wanted to do an analysis of an InfoStealer variant I discovered in the wild. The tool I went for? Responder Pro. As I said though, my demo expired and I felt a bit lost. Gone was the quick analysis. Gone was the interface. I still have Volatility and Memoryze and they certainly have their strengths but I had gotten very used to using Responder. I still have the old tried and true tools around but it's a bit of a disappointment to go back to them.

The biggest issue I have is unfortunately not technical at all. It's price - which is currently the biggest concern for us. For $9000 I could license my entire team with IDA pro and train them all in Memoryze and Volatility.

Do I recommend the Responder family of products?

Absolutely. The products have a lot of strengths including time saving techniques and easy analysis and presentation of otherwise complex data sources. For many people in the industry Responder Field Edition is more than appropriate.

Responder Pro is an entirely different beast and to be frank I feel a little naked right now.

Quickpost - new malware

New malware uploaded this evening. It's been causing problems everywhere.

Symantec Calls this TidServ.G - It poisons DHCP and DNS and redirects DNS to the Ukraine. This is the latest in DNS/DHCP poisoning malware.

I call it exemplar18 ;)

A quick word about the graphic (being a graphical person)..

The screenshot above is from HBGary's Responder Pro looking at the memory dump. Note the loop on the left hand side? That's an awesome representation of an 'if' loop that is checking if the host is running security software (anti malware). The malware will kill that software.

Gateway Malware Theory

Over time I've started developing a simple theory I'm calling the Gateway Malware Theory. Stated simply, "Simple malware leads to more complex malware, and there is no such thing as simple malware".

In more detail...

In the early days of malware we had single purpose, single focus malware that spread through a single mechanism. These days, even the simple malware is multi-vectored, multi-staged and downloads other more nefarious malware. Take Vundo for instance.

Vundo is, in other words a downloader. Once it makes its way on to a system it tends to download rogue programs or 'scareware'. On occasion I've seen it download hupigon or some other nasty program. It also infects dll's, exhausts system resources, downloads other malware and so on. According to Fireeye, it's now downloading copies of Randsom and encrypting user documents.

Vundo is "simple malware", yet it can take a mere infection from nuisance, to a fully compromised system that poses a real risk. It's what I'm calling Gateway Malware.

This leads to the Gateway Malware Theory, which goes something like this....

Simple malware infections, if not dealt with quickly, will inevitably lead to the download and installation of poorly detected malware that poses a real and true risk to organizations. The focus of any investigation of malware should be less focused on the malware and more focused on the data that is contained on, or accesible from the infected system. Therefore the first step in the investigation of malware should be data centric. If the contents of a system are unknown, then the risk, regardless of the presence of malware, can not be known or determined. As such, the presence of malware is irrelevent unless the contents of the system are known, and one must know what level of access the infected system, or user of the system has to sensitive data.


As I said I'm developing this theory still, and it's incomplete but take a look at some of the memory dumps I'm making public through my Memory snapshot project if you think you disagree. Thoughts?

Malware project updates

As I mentioned in the addendum to the last post, I had a flaw in the method I was using.

The flaw was twofold. Memory page trimming in vmware, and I wasn't allowing the malware to execute fully. I've fixed this and as a result you'll see some fairly dramatic changes in the contents of the memory snapshots.

I've uploaded a few snapshots today including:

Ackantta
Koobface
Infostealer


and I also reloaded the exemplar4 snapshot, which is an IRCbot with a few twists ;)

I'll be adding a Mebroot and Randsom variant soon. I've added a link to the blog for accessing my skydrive. Expect regular updates. If you've got specific malware you want to see in memory, email me.


Updates:

I've now uploaded 10 samples including: Waledec, Mebroot, and more.

Memory snapshot Project Part II

It appears that the memory snapshot idea has been well received so I'm in the process of uploading more snapshots to my skydrive. I think I've got a decent format now.

Under my public folder you'll see a series of exemplarX files where X is a number.

Within each directory you can expect to find the following:

about.txt - This identifies the malware and provides an md5. The binary is uploaded at offensivecomputing.net.

virustotal_.pdf - This is a .pdf file containing virustotal results for the binary.

Exemplar segments - I decided on a more universal method of compression (tar.gz) and I've split the segments using the linux split command. These segments will need to be concatenated. This can be done in linux by using the cat command. In windows, it's a copy command.

on Linux:
cat exemplar5.tar.gz.* > exemplar5.tar.gz

on Windows:
copy /b exemplar5.tar.gz.* exemplar5.tar.gz

Simply extract the .vmem from the .tar.gz file and off you go.

hashes.txt - This is a list of md5 hashes of all segmented files, the .vmem file, the .pdf, and the about.txt file.

This seems like a fairly decent model to follow though I'm open to suggestions.

I've posted a few more images and I'm in the process of creating several more.


One thing to keep in mind is that while I try to validate the execution of the malware in a virtual setting, I am fallible. If you think there's no trace of the malware in the memory dump, let me know.

Happy malware hunting.

3/21/09 addendum

A quick update.

I realized a flaw in my methodology. I didn't give the malware enough time to fully execute so I'm re-doing the exemplars.

If you downloaded exemplar4 already, I invite you to download it again.

A memory snapshot project

Some time ago, I got really tired of seeing lame attempts at proving the value of memory dumps by vendors showing that you could find "hxdef" strings in memory dumps. Today, I'd like to announce a fledgling personal project of mine. I don't yet have a name for it and it's in the very early stages but it goes something like this...

I see a lot of malware and I know there are a lot of people that don't. I also know that people want to do memory analysis but the only real source of samples is from DFRWS from 4 years ago. Here's what I'm doing...

I take 'in the wild' malware, load it up in a virtual machine, suspend the virtual machine and extract the .vmem file. I then upload the .vmem file and make it available to you, my faceless readers and the world at large. This isn't one of those "contests" where I challenge you to analyze a memory dump. Rather I am providing memory dumps of 'in the wild' malware being run in a controlled environment. Maybe this will help developers build better tools, maybe this will educate examiners, maybe this will build incident response IQ, maybe this will give students something to work with, or maybe I'll just waste some cycles providing this stuff. Time will tell.

This post is more or less a test to see if the public can access my skydrive to download the memory snapshots. Up until now, I've had issues sharing files with others. Hopefully skydrive helps with this issue.

My first snapshot is here. The file is a split .AD1 file created with FTK imager 2.5.5. You'll need to combine the segments and extract the contents. It's incredibly easy with FTK imager. The file contained within is a 7zip compressed memory image. Simply uncompress and have fun. All I ask at this point is that you let me know if you have issues, and maybe let me know if you find it valuable.

Disaster averted

It's a rare day when I have truly exciting things happen. Tonight of course was the exception. A few months ago I had a hot water heater installed by so called "professionals"..you know, the factory trained kind. I use a night rate unit that controls when the unit is active. This evening when the unit turned on all was well or so I thought. When I went to the basement to look at something, I noticed an acrid chemical smell of something metallic and plastic burning. Having had "some" experience in this arena, I could tell that it was an electrical fire. If you've never smelled an electrical fire, there's nothing else like it. The smell of the metal wire, and the plastic shielding produces a smell and taste that doesn't leave your mouth or nostrils any time soon. Anyways, I had to locate the smell. The problem with electrical fires when you're in a room full of electrical wiring, is trying to locate exactly where the smell is coming from. For this, unless you have a "hot spot" detector, you usually have to rely on the tried and true sniff test.

So, there I was sniffing around my basement like a bloodhound trying to locate the source. Finally I reached the hot water heater. When you find the source, boy...you find the source. Getting that close to the source of an electrical fire creates a bit of a gag factor but it's temporary. Needless to say I turned off the breaker and called the Fire department. The problem was contained, but I wanted to make sure there were no hot spots growing in the conduit.

Not wanting to lose the opportunity to learn, I tried to pay attention to every detail - you know, that whole "study the methods used by others" idea that I mention quite a bit. The Captain was the first on the scene. I showed him where the fire was and got out of his way. He surveyed the area, asked me a few questions and previewed the hot water heater - meaning he did a sniff test too. When the rig arrived, I went out to let them in, and I showed them where to go. They checked the area with the hot spot detector and validated my findings, then proceeded to tear apart the wiring to determine the scope of the damage. You've already seen the wiring from inside the water heater. Here's the wiring from inside the conduit.



Suffice it to say the wiring is just destroyed. The root cause was a short within the wire nut, caused by poor installation. That smoldered lump of plastic in the first picture is what used to be the wiring nut. Anyways all is well and the aftermath begins tomorrow.

Naturally, this post isn't about the fire in my hot water heater tonight. It's about incident response and a few of the things that contribute to, and separate a good outcome from a bad outcome.

1) Knowing the environment you're dealing with. In this case, this was my house. I knew what I done today that could have created the situation, I knew where each electrical item was in my basement, I knew my wiring panel, and had it labeled.

In the digital world, this is the same as knowing your organization. You need to know where your assets are, what the assets are, how they are connected and you should have an updated topological diagram.

2) Experience and awareness. I've dealt with electrical fires before and knew what the smell was. I knew that a fire was nothing I was qualified to deal with, so I called the professionals without poking around more than was necessary. I also knew that once I described the problem, answered questions and showed them the location of the fire, I should get out of their way and let them work.

In the digital world, if you're the first responder or discover the incident, if you can't solve the problem yourself and you have someone on the way, don't meddle with the system and when the IRT arrives, show them where to go, answer their questions and get out the way. Hovering when an IRT is working does not help the situation. If your assistance is required, you'll be asked to help.

3) Factory trained professionals don't always do the right thing and cut corners. As the firefighters worked they were talking to one another and discussing their findings and theorizing the root cause. The root cause was the people that installed my hot water heater.

In the digital world, consultants are well paid but don't always do the right thing. I've dealt with many cases where the root cause was the consultant's poor choices during installation. Dropping firewalls, poor password security etc. When entering an engagement with a consultant, be sure that you know what you're getting.

These are just a few of the things that you should be aware of in the world of incident response. The biggest lessons of the night for anyone out there that has an Incident Response Team at your disposal is:

• If you are unsure, call the trained people that do know, before you do anything.
• There's no shame in admitting you don't know everything and can't solve the problem.
• If you know something is out of the ordinary, call quickly.

A safe evening to all.