Disaster averted

It's a rare day when I have truly exciting things happen. Tonight of course was the exception. A few months ago I had a hot water heater installed by so called "professionals"..you know, the factory trained kind. I use a night rate unit that controls when the unit is active. This evening when the unit turned on all was well or so I thought. When I went to the basement to look at something, I noticed an acrid chemical smell of something metallic and plastic burning. Having had "some" experience in this arena, I could tell that it was an electrical fire. If you've never smelled an electrical fire, there's nothing else like it. The smell of the metal wire, and the plastic shielding produces a smell and taste that doesn't leave your mouth or nostrils any time soon. Anyways, I had to locate the smell. The problem with electrical fires when you're in a room full of electrical wiring, is trying to locate exactly where the smell is coming from. For this, unless you have a "hot spot" detector, you usually have to rely on the tried and true sniff test.

So, there I was sniffing around my basement like a bloodhound trying to locate the source. Finally I reached the hot water heater. When you find the source, boy...you find the source. Getting that close to the source of an electrical fire creates a bit of a gag factor but it's temporary. Needless to say I turned off the breaker and called the Fire department. The problem was contained, but I wanted to make sure there were no hot spots growing in the conduit.

Not wanting to lose the opportunity to learn, I tried to pay attention to every detail - you know, that whole "study the methods used by others" idea that I mention quite a bit. The Captain was the first on the scene. I showed him where the fire was and got out of his way. He surveyed the area, asked me a few questions and previewed the hot water heater - meaning he did a sniff test too. When the rig arrived, I went out to let them in, and I showed them where to go. They checked the area with the hot spot detector and validated my findings, then proceeded to tear apart the wiring to determine the scope of the damage. You've already seen the wiring from inside the water heater. Here's the wiring from inside the conduit.



Suffice it to say the wiring is just destroyed. The root cause was a short within the wire nut, caused by poor installation. That smoldered lump of plastic in the first picture is what used to be the wiring nut. Anyways all is well and the aftermath begins tomorrow.

Naturally, this post isn't about the fire in my hot water heater tonight. It's about incident response and a few of the things that contribute to, and separate a good outcome from a bad outcome.

1) Knowing the environment you're dealing with. In this case, this was my house. I knew what I done today that could have created the situation, I knew where each electrical item was in my basement, I knew my wiring panel, and had it labeled.

In the digital world, this is the same as knowing your organization. You need to know where your assets are, what the assets are, how they are connected and you should have an updated topological diagram.

2) Experience and awareness. I've dealt with electrical fires before and knew what the smell was. I knew that a fire was nothing I was qualified to deal with, so I called the professionals without poking around more than was necessary. I also knew that once I described the problem, answered questions and showed them the location of the fire, I should get out of their way and let them work.

In the digital world, if you're the first responder or discover the incident, if you can't solve the problem yourself and you have someone on the way, don't meddle with the system and when the IRT arrives, show them where to go, answer their questions and get out the way. Hovering when an IRT is working does not help the situation. If your assistance is required, you'll be asked to help.

3) Factory trained professionals don't always do the right thing and cut corners. As the firefighters worked they were talking to one another and discussing their findings and theorizing the root cause. The root cause was the people that installed my hot water heater.

In the digital world, consultants are well paid but don't always do the right thing. I've dealt with many cases where the root cause was the consultant's poor choices during installation. Dropping firewalls, poor password security etc. When entering an engagement with a consultant, be sure that you know what you're getting.

These are just a few of the things that you should be aware of in the world of incident response. The biggest lessons of the night for anyone out there that has an Incident Response Team at your disposal is:

• If you are unsure, call the trained people that do know, before you do anything.
• There's no shame in admitting you don't know everything and can't solve the problem.
• If you know something is out of the ordinary, call quickly.

A safe evening to all.