There’s an old saying in medicine, “when you hear hoof beats, think horses, not zebras.” Contrary to what one might think after watching House the truth is, when you get presented a set of symptoms, you start with the most likely, well because it is most likely! But as House illustrates, sometimes it can be the unlikely.
In First Aid, especially wilderness or backcountry medicine, there’s an acronym that is often used called SAMPLE. This is a mnemonic to help rescuers remember what data to gather:
- S – Signs/Symptoms – What do you see, observe? (i.e. what’s going on).
- A – Allergies – Perhaps the problem is an allergic reaction, or they might be allergic to whatever drug you want to give them.
- M – Medicines – What medications are they on? Perhaps they’re diabetic and haven’t taken their insulin, or they’re on an anti-seizure medicine and need some.
- P – Past, pertinent medical history. You don’t care they broke their ankle when they were 5. But perhaps they just underwent surgery a few weeks ago? Or perhaps they have a history of dislocating their shoulder.
- L – Last oral intake. Have they eaten or drunk anything recently. This will drive your decision tree in a number of ways.
- E – Events leading up to the injury. Were they climbing to the top of the cliff and fell, or did they simply collapse at the bottom? The former may suggest you look for a possible spinal injury, the latter probably indicates something else.
I’m reminded of this because of how I spent my Presidents’ Day. I woke up and checked a few emails and noticed that two I were expecting from a client’s servers never arrived. I logged in to see what was going on. It turns out there was nothing going on. No, not as in, “nothing wrong going on” but more as in “nothing at all was happening, the databases weren’t operating right.” The alert system could connect to the database server, so no alerts had been sent, but actually accessing several of the databases, including the master resulted in errors.
And what was worse, was the DR server was exhibiting similar symptoms!
So modifying SAMPLE a bit:
- S – Signs/Symptoms – Well, databases are throwing corruption errors on two servers. This was extended to the ERRORLOG files on both servers.
- A – Allergies – Well, servers don’t have allergies, but how about known bugs? That’s close enough. Nope, nothing that seems to apply here.
- M – Medicines – I’ll call this antivirus software and <redacted>. (For client privacy reasons I can’t specify the other piece of software I want to specify, but I’ll come back to it.)
- P – Past, pertinent medical history. Nothing, these servers had been running great up until now. One has been in production for over 2 years, the other, up for about 2 months, being brought up as a DR box for the first.
- L – Last oral intake. Let’s make this last data intake. Due to forensics, we determined the corruption on both servers occurred around 3:00 AM EST. Checking our logs, jobs, and other processes, there’s nothing special about the data the primary server took in at this time. If anything, disk I/O was lower than average. And, fortunately, we can easily recreate any data that was sent to the server after the failure.
- E – Events leading up to the injury. This is where things get interesting.
- There were some zero-day patches applied to both servers over the weekend.
- On Saturday, I had finally setup log-shipping between the two servers
So, we’ve got three possibilities, well four really.
- The zero day patches caused an issue about 48 hours later. Possible, but unlikely, given the client has about 1600 servers that were also patched and have not had issues.
- Log-shipping somehow caused problems. But again, the new log-shipping setup had run for about 36 hours without issue. And, best we can tell, the corruption occurred on the secondary BEFORE the primary. And log-shipping doesn’t apply to the Master database or the ERRORLOG file.
- Some unknown interaction. This I think is the most likely and is where the <redacted> from the M above comes into play.
- Pure Random, and it’ll never happen again. I hate this option because it just leaves me awake at night. This I added only for completeness.
Without going into detail, our current theory is that some weird interaction between <redacted> and log-shipping is our cause. Of course the vendor of <redacted> is going to deny this (and has) but it’s the only combination of factors that seems to explain everything. (I’ve left out a number of additional details that also helped us get to this conclusion).
So for now, we’ve disabled log-shipping and are going to make some changes to the environment before we try log-shipping again.
Normally I think horses, but we might have a herd of zebras on this one. And ironically, setting up for DR, may have actually caused a Sev 1 outage. So the ounce of prevention here may not have been worth it!
And who said my Wilderness First Aid wouldn’t come in handy?