Backups Are Useless

I’m going to take a controversial stand and argue that backups are useless.

Over the last few months I’ve worked with a client of mine to test their Disaster Recovery procedure for one of their major in-house applications. This involved multiple several-hour meetings with anywhere from 5 to 10 people at each meeting, sometimes more. Each hour probably cost the client $1000s of dollars. The cost of running these meetings and tests probably cost the client well over $100K.

This is ignoring the costs of the associated hardware, the power for the backup datacenter, the cost of heating and cooling, and of course the licensing. I wouldn’t be surprised if they easily spend more than $1 Million a year in backups and the like.

And for what? A fairly low probability event?

I mean sure, if their system failed and they had no Disaster Recovery plan it could cost them 10s of millions of dollars in business and perhaps even end up putting 100s of people out of work. But they’d find other jobs. In the meantime, all that money spent on backups could have been spent on other things like lunches for for the employees (and maybe a pizza or two for select consultants). Think of the boost to the pizza economy that would have been!

So, don’t do backups.

Oh and don’t wear a mask. They’re hot, sweaty, and really not even .1% of the US population has died. And sure, you might get COVID-19, but you’ll probably survive. Sure, you might have some cognitive long-term issues, but hey, that’s sort of like the employees at my client above who, if the company went under, could simply find another job. I mean it’s not a big deal. Amirite?

Now let’s be serious. If a DBA came in to your business and said not to bother doing backups, you’d probably laugh at them. Do backups. And of course wear a mask. There is so much evidence it makes a difference. And, socially distance for now. And reconsider large family gatherings for the next month or two, if only to help increase the odds that you can have such a gathering a year from now.

Part of this post was prompted by a question on Quora from a user asking how to recover their database if they didn’t have a backup. I hated to tell them that it might be too late and there was quite likely little they could do. And I’ve read too many heart-wrenching stories from nurses who have had to hold the hand of a dying patient because they thought Covid was no big deal or a hoax. So, please, take precautions. Even if nothing happens to you, it may happen to those close to you.

That said, I will repeat an adage about backups I heard a few SQL Saturdays ago: “Backups don’t matter, restores do!” So do backups, but restore them every once in awhile to make sure that they actually work!

For the record, with my client, not only did the official DR test run go smoothly, we beat our RTO and RPO by huge margins. If disaster strikes, it’s highly likely this customer will weather it without threatening the future of the company.

Caving and SQL

Longtime readers know that I spend a lot of my time talking about and teaching caving, more specifically cave rescue, and SQL Server, more specifically the operations side. While in some ways they are very different, there are areas where they overlap. In fact I wrote a book taking lessons from both, and airplane crashes to talk about IT Disaster Management.

Last week is a week where both had an overlap. One of the grottoes in the NSS (think like a SQL User Group) sponsored a talk on Diversity and Inclusion in the caving community. The next day, SQL Pass had a virtual panel on the exact same subject.


Let me start with saying that one thing I appreciate about both communities is that they will welcome pretty much anyone. You show up and ask to be involved and someone will generally point you in the right direction.  In fact several years ago, I heard an Oracle DBA mention how different the SQL community was from his Oracle experience, and how welcoming and sharing we could be.

This is true in the caving community. I recall an incident decades ago where someone from out of town called up a caving friend he found in the NSS memberhsip manual and said, “hey, I hear you go caving every Friday, can I join you?” The answer was of course yes.  I know I can go many places in this country, look up a caver and instantly be pointed to a great restaurant, some great caves and even possibly some crash space to sleep.

So let’s be clear, BOTH communities are very welcoming.

And I hear that a lot when the topic of diversity and inclusion comes along. “Oh we welcome anyone. They just have to ask.”


Well, there’s two issues there and they’re similar in both communities. The less obvious one is that often anyone is welcome, but after that, there’s barriers, some obvious, some less so. Newcomers start to hear the subtle comments, the subtle behaviors. For example, in caving, modesty is often not a big deal. After crawling out of a wet muddy hole, you may think nothing of tearing off your clothes in the parking lot and changing. Perhaps you’re standing behind a car door but that’s about it. It’s second nature, it’s not big deal. But imagine now that you’re the only woman in that group. Sure, you were welcomed into the fold and had a blast caving, how comfortable are you with this sudden lack of modesty? Or you’re a man, but come from a cultural or religious background where modesty is a high premium?

In the SQL world, no one is getting naked in the datacenters (I hope). But, it can be subtle things there too. “Hey dudes, you all want to go out for drinks?” Now many folks will argue, “dudes is gender neutral”. And I think in most cases it’s INTENDED to be. But, turn around and ask them, “are you attracted to dudes?” and suddenly you see there is still a gender attached.  There’s other behaviors to. There’s the classic case of when a manager switched email signatures with one of his reports and how the attitudes of the customers changed, simply based on whose signature was on the email.

So yes, both groups definitely can WELCOME new folks and folks outside of the majority, but do the folks they welcome remain welcomed? From talking to people who aren’t in the majority, the answer I often get is “not much.”

An Interlude

“But Greg, I know….” insert BIPOC or woman or other member of a minority.  “They’re a great DBA” or “They’re a great caver! Really active in the community.”  And you’re right. But you’re also seeing the survivorship bias. In some cases, they did find themselves in a more welcoming space that continued to be welcoming. In some cases you’re seeing the ones who forged on anyway. But think about it, half our population is made up of women. Why aren’t 1/2 our DBAs?  In fact, the number of women in IT is declining! And if you consider the number of women in high school or college who express an interest in IT and compare it to those in in their 30s, you’ll find the number drops. Women are welcome, until they’re not.

In the caving community during an on-line discussion where people of color were speaking up about the barriers they faced, one person, a white male basically said, “there’s no racism in caving, we’ll welcome anyone.”  A POC pointed out that “as a black man in the South, trust me, I do NOT feel safe walking through a field to a cave.”  The white man continued to say, “sure, but there’s no racism in caving” completely dismissing the other responder’s concerns.

There’s Still More…

The final point I want to make however is that “we welcome people” is a necessary, but not sufficient step. Yes, I will say pretty much every caver I know will welcome anyone who shows an interest. But that’s not enough. For one thing, for many communities, simply enjoying the outdoors is something that’s not a large part of their cultural.  This may mean that they’re not even aware that caving is a possibility. Or that even if it is, they may not know how to reach out and find someone to take them caving.

Even if they overcome that hurdle, while caving can be done on the cheap, there is still the matter of getting some clothing, a helmet, some lights. There’s the matter of getting TO the cave.

In the SQL world, yes anyone is welcome to a SQL Saturday, but what if they don’t have a car? Is mass transit an option? What if they are hearing impaired? (I’ve tried unsuccessfully 2 years in a row to try to provide an ASL interpreter for our local SQL Saturday. I’m going to keep trying). What if they’re a single parent? During the work week they may have school and daycare options, but that may not be possible for a SQL Saturday or even an afterhours event. I even had something pointed out to me, during my talk on how to present, that someone in the audience had not realized up until I mentioned it, that I was using a laser pointer. Why? Because they were colorblind and never saw the red dot. It was something that I, a non-colorblind person had never even considered. And now I wonder, how many other colorblind folks had the same issue, but never said anything?

In Conclusion

It’s easy and honestly tempting to say, “hey, we welcome anyone” and think that’s all there is to it. The truth is, it takes a LOT more than that. If nothing else, if you’re like me, an older, cis-het white male, take the time to sit in on various diversity panels and LISTEN. If you’re invited to ask questions or participate, do so, but in a way that acknowledges your position. Try not to project your experiences on to another. Only once have I avoided a field to get to a cave, because the farmer kept his bull there. But I should not project MY lack of fear about crossing a field onto members of the community who HAVE experienced that.

Listen for barriers and work to remove them. Believe others when they mention a barrier. They may not be barriers for you, but they are for others. When you can, try to remove them BEFORE others bring them up. Don’t assume a barrier doesn’t exist because no one mentions it. Don’t say, “is it ok if I use a red laser pointer?” because you’re now putting a colorblind person on the spot and singling them out. That will discourage them. For example find a “software” pointer (on my list of things to do) that will highlight items directly on the screen. This also works great for large rooms where there may be multiple projection screens in use.

If caving, don’t just assume, “oh folks know how to find us” reach out to community groups and ask them if they’re interested and offer to help. (note I did try this this year, but never heard back and because of the impact of Covid, am waiting until next year to try again.)

Don’t take offense. Unless someone says, “hey, Greg, you know you do…” they’re not talking about you specifically, but about an entire system. And no one is expecting you to personally fix the entire system, but simply to work to improve it where you can. It’s a team effort. That said, maybe you do get called out. I had a friend call me out on a tweet I made. She did so privately. And she did so because, she knew I’d listen. I appreciated that. She recognized I was human and I make mistakes and that given the chance, I’ll listen and learn. How can one take offense at that? I saw it has a sign of caring.

Finally realize, none of us are perfect, but we can always strive to do better.

So, today give some thought about how you can not only claim your community, whatever it may be, is welcoming, but what efforts you can make to ensure it is.


On a separate note, check out my latest writing for Red-Gate, part II on Parameters in PowerShell.

It’s too late?

I want to start with a sobering thought. It’s too late to contain this pandemic. I’m watching the news as slowly more and more states in the US issue versions of “shelter in place” or “stay at home” orders. But I think in most cases, it’s too late. The virus has probably already spread so much that self-isolation won’t be nearly as effective as it would have been had the states issued the same orders a week or two earlier. That said, it’s most likely still better than doing nothing.

Human beings at times are lousy with risk analysis. If a risk is immediate, we can react well, but the longer it stretches out or the further away it is, the harder it is to get people to react. Almost any climate scientist who has studied anthropogenic global warming has known for a decade or more we have a problem and we have a very quickly narrowing window for solving it, and the longer we wait, the harder it will become.

Yet too many of us put off the problem for another day.

So it is with the Covid-19 virus. “Oh we don’t have to lock down just yet, let’s wait another day.” And I’ll admit, sitting in the state that is the center of the virus outbreak here in the US, I’m tempted to say, “25,000 isn’t TOO bad, we can manage that.”  But that’s the lizard part of my brain reacting. It’s the emotional part. Then I kick in the rational part. If we use one of the numbers bandied about, doubling every 4 days, that means by this weekend, in New York State alone, it will be 50,000. By April 1st, 100,000. By the end of April, it could be the entire state.  Those numbers are hard to comprehend.

That said, I’m also hopeful. Modelling pandemics is pretty much pure math, but reality is more complex and often luck can play a huge factor. Let me try to explain.

First, we need to heed the words of experts like Dr. Fauci and others who are basing their remarks and recommendations on the inexorable exponential rise in expected infections. They are giving basically the worst case scenario if their recommendations are followed. And that’s proper. That’s really what you have to plan for.

Let me take a little side trip and mention a cave rescue in Vermont several years ago. By the time I had gotten the call to show up and to call out other rescues, the injured party had been in the cave for several hours. I didn’t know much about the extent of their injuries other than it was a fall and that it was in a Vermont cave, which almost certainly meant operating in tight quarters. I grabbed a box of Freihofer cookies, a lawn chair (my fellow cave rescuers will understand the reference), a contact list of other potential rescuers, and my son. While I drove, he’d read off a name and I’d say “yes call” or “Nope, next name.”  On the hour plus drive to the rescue we managed to contact at least two other people who could get there. (It turns out, as I surmised, several of the folks I wanted to call were members of the original caving party.)

Once there, my son and I were driven partway to the cave entrance and trudged the rest of the way. I talked with the folks on the scene to gather information and then dressed to go into the cave to gather first hand information. I still hadn’t gained too much information other than to know it was potentially shaping up to be a serious rescue. The person had been climbing a cable ladder when they fell and injured themselves. This meant, based on the information at hand, a worst case scenario of an evac through tight passages with the patient in a SKED stretcher.  I was playing the role of Dr. Fauci at that point, preparing for the worse based on the information I had.

Fortunately, literally at the moment I was about to enter the cave, one of the members of the original caving party crawled out and said, “he’s right behind me, he’ll be out in a minute or so.”  It turns out his injuries were fairly minor and with the members of his own caving party, he was able to get out of the cave under his own power.

I got back to Incident Command about an hour later and was informed, “oh, by the way, you’ve got at least 3 cavers who showed up to help. We held them at the bottom of the road. What should we tell them?”  My answer was simply, “Thanks and to go home.”

I relate this story not so much to talk about cave rescue specifically but to point out that even when planning for the worst, you may get a lucky break. But you can’t rely on them. Let me give an alternate scenario. Let’s say I had not called out the other rescuers and had gotten to the cave and crawled in, realized the situation was a worst case scenario, crawled back out and then initiated a call-out. It would have at that point probably meant at least an extra 90 minutes before the extra resources would have been on the scene. It would have meant the patient was exposed to hypothermic condition for another 90 minutes. It would have meant 90 more minutes of pain. It would have meant fewer brains working to solve the problem.

Getting back to Covid-19. Will we get lucky? I don’t know. I actually suspect we might. One “advantage” of an increasing population of sick people is we can better model it and we can also perform more drug trials. We may discover certain populations react differently to the disease than others and be able to incorporate that into the treatment plan. I don’t know. But I do know, we need to plan for the worst, and hope for a bit of luck. In the meantime, hunker down and let’s flatten the curve.

And if you’ve read this far and want to know how to make some pita bread, I’ll let you in on the two secrets I’ve learned: VERY hot oven (I typically bake mine at about 500-550F for 2 minutes on 1 side, and 1 minute on the other) and roll it out thinner than you might think.

Life in a Time of Coronavirus

With apologies to Gabriel García Márquez.

Life in the past week has definitely take a turn. We’ve gone from, “this might be bad” to losing about 30% of the market value of the stock market, and basically the country is shutting down.

I’d like to quote R.E.M. and say “It’s end of the world as we know it, and I feel fine” but the truth is more complex. Physically, I feel fine. I don’t think I have the Coronavirus, but of course I can’t say for sure.  I’m not really worried about myself. I’ve been working from home for years and so already had reduced the vector of working in crowded offices. But, of course my wife has been working in an office (though tomorrow her job will become “work from home”) and my son just came home from college for a break of now unknown length and my daughter’s school started a mandatory closure with some sort of undefined “remote teaching”. It’s still unclear what teachers will do what, but I hope by the end of the week they’ll be able to have a more robust teaching rubric in place.

I’ve been thinking a lot about toilet paper lately. We picked up some more rolls, just in case, but we’re far from hoarding (unlike the story of one family of four I read about on Facebook that apparently was caught trying to bypass the rules at a supermarket and buying 16 packs of 24 rolls of toilet paper: yes, they were trying to buy a total of 384 rolls of toilet paper in one day!)

But I’d by lying if I didn’t say there was a certain about of dread on my mind. Things are going to get worse before they get better.  I’ve been monitoring a site out of Johns Hopkins that seems to have accurate and fairly up to date data on the spread of the Coronavirus. The numbers aren’t great.  If the spread continues according to some models, we could be looking at a death toll in the US of 500,000-1 million or more.  These are staggering numbers. Now, to be clear, with social distancing and other measures, I have reason to hope the numbers will be 1/10th of those numbers, but even that’s a good sized number.

And of course there’s the aforementioned drop in the value of the stock market, and I honestly think it will only get worse.  If the worse case death tolls come in, we’re looking at a fundamental shake-up of our economy. Couple that with some of the possible mid-term effects as people face bankruptcy due to income loss and longer term changes that may result as a result of social changes. For example I’ve seen one headline suggesting that cinemas may simply not make a comeback after people stop going and just watch them at home. This is all scary and worrisome. And it’s bigger than toilet paper.

But more importantly, I’ve been thinking about families. I’ve written about this before and I’ve mentioned #SQLFamily. With Social Distancing being the current buzzwords, it’s put a damper on getting together with people. I know it’s so tempting to call up some friends and say, “hey, let’s get together and play games” but that sort of negates the point of social distancing.

My families, virtual and my biological one give me hope and cause to celebrate. On the RPI based chat program I use, Lily, it’s been great to see how one whole discussion is pretty much focused on a fact-based exchange of ideas. Even those of us on separate ends of the political spectrum are basically exchanging facts and mostly keeping emotions in check. This has been a useful place to learn a lot.

And on Twitter and elsewhere, it’s been remarkable to see #SQLFamily come together. Last Friday, and coming again this Friday, one of our members hosted a Zoom Chat where #SQLFamily members could just hang out and chat. Yeah, we talked about TP, and power outages in Johannesburg, and other topics, but it was mostly fun small talk. It was a reminder, that there are real people behind the Twitter handles and tweets. I’ve seen my #SQLFamily members send tweets about the success of their own family members and of their own hopes and fears.

Having gone to a technical college, and usually surrounded by folks who are self-identified geeks, sometimes it can be tempting to think we’re all just emotionless people driven by facts and data, human version of Star Trek’s Mr. Spock or Data, but the reality is, the families I’ve surrounded myself with are an amazing, resilient, group of people. And I think that’s probably true of most of us. It’s tempting to think “my group is special” and in some ways every group is, but at the end of the day, I think what unites us is greater than what separates us. Yes, fortunately I think my virtual families tend to make decisions a bit more based on facts than some groups, but we still share the same humanity deep down.

So, is there a tinge of dread on the horizon of my mind, yes. But I also see the sun rays of hope peaking above the clouds and know that the next weeks, months and possibly year or more will be rough. Some of us may lose loved ones. And I hate to type that, but it’s true. But, life will go on and we’ll find a new normal and we will do so by maintaining our relationships, physical and virtual.

My advice, during these times, reach out, expand your virtual relationships. Find hope and happiness where you can and share your fear, sadness, and sorrow when you need to.

Do small acts of kindness.

And I make the following offer:

If you’re down and need to talk or even just a cheer-up hit me up on twitter @stridergdm or if you know me on Facebook, private message me. If you want to join me on Lily, let me know, I’ll set you up with an account and a client. It’s mostly RPI folks, but not exclusively. If you really need it, we can even to a Zoom chat to talk about anything, from the role of the little known Saturn IV stage to talking about the Hudson River Chain at West Point during the Revolutionary War to recipes for air fryers, I’m there.

I’m going to close with a bit of randomness, because, well I think we need it.

A random cow sighting at a local Walmart.

A random cow sighting at a local Walmart.


An Ounce of Prevention?

There’s an old saying in medicine, “when you hear hoof beats, think horses, not zebras.”  Contrary to what one might think after watching House the truth is, when you get presented a set of symptoms, you start with the most likely, well because it is most likely! But as House illustrates, sometimes it can be the unlikely.

In First Aid, especially wilderness or backcountry medicine, there’s an acronym that is often used called SAMPLE. This is a mnemonic to help rescuers remember what data to gather:

  • S – Signs/Symptoms – What do you see, observe? (i.e. what’s going on).
  • A – Allergies – Perhaps the problem is an allergic reaction, or they might be allergic to whatever drug you want to give them.
  • M – Medicines – What medications are they on? Perhaps they’re diabetic and haven’t taken their insulin, or they’re on an anti-seizure medicine and need some.
  • – Past, pertinent medical history. You don’t care they broke their ankle when they were 5. But perhaps they just underwent surgery a few weeks ago? Or perhaps they have a history of dislocating their shoulder.
  • L – Last oral intake. Have they eaten or drunk anything recently. This will drive your decision tree in a number of ways.
  • E – Events leading up to the injury.  Were they climbing to the top of the cliff and fell, or did they simply collapse at the bottom? The former may suggest you look for a possible spinal injury, the latter probably indicates something else.

I’m reminded of this because of how I spent my Presidents’ Day.  I woke up and checked a few emails and noticed that two I were expecting from a client’s servers never arrived. I logged in to see what was going on.  It turns out there was nothing going on. No, not as in, “nothing wrong going on” but more as in “nothing at all was happening, the databases weren’t operating right.”  The alert system could connect to the database server, so no alerts had been sent, but actually accessing several of the databases, including the master resulted in errors.

Master.mdf corruption

Not an error you want to wake up to!

And what was worse, was the DR server was exhibiting similar symptoms!

So modifying SAMPLE a bit:

  • S – Signs/Symptoms – Well, databases are throwing corruption errors on two servers.  This was extended to the ERRORLOG files on both servers.
  • A – Allergies – Well, servers don’t have allergies, but how about known bugs? That’s close enough.  Nope, nothing that seems to apply here.
  • M – Medicines – I’ll call this antivirus software and <redacted>. (For client privacy reasons I can’t specify the other piece of software I want to specify, but I’ll come back to it.)
  • – Past, pertinent medical history.  Nothing, these servers had been running great up until now. One has been in production for over 2 years, the other, up for about 2 months, being brought up as a DR box for the first.
  • L – Last oral intake. Let’s make this last data intake. Due to forensics, we determined the corruption on both servers occurred around 3:00 AM EST.  Checking our logs, jobs, and other processes, there’s nothing special about the data the primary server took in at this time. If anything, disk I/O was lower than average.  And, fortunately, we can easily recreate any data that was sent to the server after the failure.
  • E – Events leading up to the injury. This is where things get interesting.
    • There were some zero-day patches applied to both servers over the weekend.
    • On Saturday, I had finally setup log-shipping between the two servers

So, we’ve got three possibilities, well four really.

  1. The zero day patches caused an issue about 48 hours later.  Possible, but unlikely, given the client has about 1600 servers that were also patched and have not had issues.
  2. Log-shipping somehow caused problems.  But again, the new log-shipping setup had run for about 36 hours without issue. And, best we can tell, the corruption occurred on the secondary BEFORE the primary. And log-shipping doesn’t apply to the Master database or the ERRORLOG file.
  3. Some unknown interaction.  This I think is the most likely and is where the <redacted> from the M above comes into play.
  4. Pure Random, and it’ll never happen again. I hate this option because it just leaves me awake at night.  This I added only for completeness.

Without going into detail, our current theory is that some weird interaction between <redacted> and log-shipping is our cause. Of course the vendor of <redacted> is going to deny this (and has) but it’s the only combination of factors that seems to explain everything. (I’ve left out a number of additional details that also helped us get to this conclusion).

So for now, we’ve disabled log-shipping and are going to make some changes to the environment before we try log-shipping again.

Normally I think horses, but we might have a herd of zebras on this one. And ironically, setting up for DR, may have actually caused a Sev 1 outage. So the ounce of prevention here may not have been worth it!

And who said my Wilderness First Aid wouldn’t come in handy?



Brake (sic) it to Fix it!

Last week I wrote about getting my brakes fixed. Turns out I made the right choice, besides shoes and pads, I needed new calipers.  Replacing them was a bigger job than I would have wanted to deal with. Of course it cost a bit more than I preferred, but I figure being able to make my car stop is a good thing. I had actually suspected I’d need new calipers because one of the signs of my brakes needing to be replaced was the terrible grinding and shrieking sound my left front brake was making. That’s never a good sign.

As a result I started to try to brake less if I could help it. And I cringed a bit every time I did brake. It became very much a Pavlovian response.

About two weeks ago, a colleague of mine said, “Hey, did you notice the number of errors for that process went way up?” I had to admit, nope, I had not. I had stopped looking at the emails in detail. They were for an ETL process that I had written well over a year ago. About 6 months ago, due to new, and bad, data being put into the source system, the ETL started to have about a half dozen rows it couldn’t process.  As designed, it sent out an email to the critical parties and I received copies.  We talked about the errors and decided that they weren’t worth tracking down at the time.  I objected because I figure if you have an error, you really should fix it. But I got outvoted and figured it wasn’t my concern at that point. As a result, we simply accepted that we’d get an email every morning with a list of rows in error.

But, as my colleague pointed out, about 3 months ago, the number of errors had gone up. This time it wasn’t about a half dozen, it was close to 300. And no one had noticed.  We had become so used to the error emails, our Pavlovian response was to ignore them.

But, this number was too large to ignore. I ended up doing two things. The first, and one I could deploy without jumping through hoops was to update the error email. Instead of simply showing the rows of errors, it now included a query that placed a table at the top that showed how many errors and in which tables. This was much more effective because now a single glance can easily show if the number of errors has increased or gone down (if we get no email that means we’ve eliminated all the errors, the ultimate goal in my mind.)

I was able to track down the bulk of the 300 new errors to a data dictionary disagreement (everyone raise their hands who has had a customer tell you one thing about data only to discover that really the details are different) that popped up when a large amount of new data was added to the source system.

I’ve since deployed that change to the DEV environment and now that we’re out of the end of the month code freeze for this particular product, will be deploying to production this week.

Hopefully though the parties that really care about the data will then start paying attention to the new email and squawking when they see a change in the number of bad rows.

In the meantime, it’s going to take me awhile to stop cringing every time I press my brakes. They no longer make any bad sounds and I like that, but I’m not used to to absence of grinding noises again.  Yet. In both cases, I and my client had accepted the normalization of deviance and internalized it.

I wrote most of this post in my head last week while remembering some other past events that are in part related to the same concept.  As a result this post is dedicated to the 17 American Astronauts who have perished directly in the service of the space program (not to diminish the loss of the others who died in other ways).

  • Apollo 1 – January 27th, 1967
  • Challenger – STS-51L – January 28th, 1986
  • Columbia – STS-107 – Launch January 16th, 2003, break-up, February 1st, 2003.



He’s dying Jim!

Less than a minute after the mountain biker blew by me on the trail I heard the wail. It was scary. I raced forward with my hiking partner and very quickly came upon the accident scene. Thomas was laying in a crumpled mess, his $1500 mountain bike further down the trail with one more bend in the frame than the manufacturer had created it with.  This didn’t look good.

The hillside was steep and Thomas was on his side. I dropped my pack and went to backside of Thomas. Quickly my training kicked in. I introduced myself and asked him his name. His response re-assured me. He was breathing and had a pulse. And in medical terms, he appeared to be alert and oriented times three.

I put on my gloves (yes, I do carry nitrile or latex or material gloves pretty much anyplace I go, you should too!)  I then moved to his backside and palpated his head. So far, so good. No blood or cerebral-spinal fluid.  Since he was on his side, it was an ideal (if one could call his situation that) position for me to check his spine. Working down, so far so good until I got to the top of his lumbar portion. There I felt something very wrong.  Ok, shit just got real.  Upper torso, so far so good. Lower torso, right side, a reaction to pain. Barely noticeable, but definitely some mass internally. Again, not good. I continued down his legs. I got to his feet and normally I’d have saved this for the secondary survey, but since I already had a bad feeling and this wouldn’t take very long, I asked him to press his toes against my hands like he was pressing the gas. Nothing. I asked him to pull up his toes, again nothing.  This was not good. Same lack of reaction on the other side.  I could hear the panic in his voice, “why I can’t I move my legs?”

In all my past training they always taught us, “never lie to the patient. But also remember you’re not a doctor.” So I was honest. “Look, it could be a lot of things. I’m not a doctor so I don’t want to speculate. We’ll leave that to the professionals.” But deep down I knew. Bad tumble off a bike, bad position of a lumbar vertebra, and lack of sensation distal to that all added up to some sort of spinal injury. Would it be permanent, I had no idea.

I also checked his right arm since it was immediately available and this time came up with blood and point pain over the radius/ulna.  This matched what the accident most likely was.  He had hit a rock or something and wrapped his right side around a tree, breaking his arm, damaging his spine and most likely causing internal bleeding.

But I still had the left side to check.

With my partner’s help, we got a ground pad behind him and then rolled him very gently onto it. It’s always a risk moving a patient with apparent trauma like this but we needed to get him isolated from the could ground which was stealing his body heat and I needed to check for injuries on the left side.

Fortunately, this was the only bright news. A thorough check of his left side showed no apparent trauma. But, his shivering was getting worse and his mental state was decomposing. Whereas he was had previously been alert and oriented to who he was, where he was and approximately the time, now he kept asking the time.  Taking a set of vitals, things were not what one would like to get.

My partner was writing down all this information and giving me gear as needed. We had gotten the groundpad under him and clothing on top, but we needed to do more.  At this point, since we confirmed he wasn’t about to bleed out, we worked on splinting his arm. Providing traction in-line proved to be a bit painful at first, but ultimately gave him some pain relief and temporarily solved that particular issue. We did as much as we could in the back-country.

I took another set of vitals, and the numbers were a wee bit worse. In addition, Thomas was now wondering where he was. I repalpated his lower right abdomen and got an increased pain response and the firm area was larger. This was a very worrying sign.

While doing this, my partner ran down the trail with a copy of his notes until he got cell service and called 911. He gave them the details and then came back.  At this point, with his help we decided to get Thomas into a bivy sack to help keep him warm.  This took some effort since Thomas was bigger than either of us, fairly muscular and we were doing out best to protect his spine. But eventually we got it around him. Between this and some liquid “squirt” we were able to give him, his pending hypothermia appeared to stabilize and eventually improve.

But his vitals continued to degrade and the pain and mass in lower right his abdomen continued to get worse. He was dying and there was nothing I could do about it.

Well there was one thing I could do. I looked at Thomas and said, “well I think that’s it. Did I miss anything or do you think we got this exercise covered?”

He looked up and smiled, “Nope, I think you got it.  By the way, the biv sack really did help a lot. I was actually starting to get cold for real.” We removed the biv sack and he remarked, “Wow, you really did have a lot of warm stuff on me.”

Now, fortunately, all this had been an exercise, part of a SOLO Wilderness First Aid class I was taking over the weekend in order to renew my certificate.  The scenario was completely made up. But, I have to say, the feeling of helplessness was real.

Strangely though I’d say that was a good thing. For any skill we want to maintain competency in, we need to practice. Fortunately, I haven’t come across a crumpled mountain biker and most if not all back-country medical emergencies I’ve encountered have basically been fairly simple (an abrasion here, a blister there, or most commonly, mild hypothermia). But, continual practice does help. When arriving at the staged scene, I knew what I wanted to do and I knew how I wanted to do it and how to do it. The years of training and practice came back very easily. I knew how to do a primary survey and what I wanted to look for. I knew what vitals I needed and what trends I wanted.  There wasn’t much searching for knowledge, it bubbled up as needed. Practice really does in a sense “make perfect.”

And, even knowing that my mock patient most likely had internal bleeding that was leading to hypovolemic shock was good to know. Knowing that there was very little I could do if this was a for-real in the back-country was scary, but also strangely reassuring. I was confident that I had done all that I could reasonably do. And sometimes that may have to be enough.

I’ve talked previously about training as one fights. This should be true in any situation you may find yourself in: caving, the back-country, or even something as mundane as being a DBA. When’s the last time you practiced restoring a backup or doing a failover test?

Practice may or may not make perfect, but it does provide confidence.

P.S. if you’re in the Hampton Roads area tomorrow night (October 16th) come check me out speaking on System Databases at the Hampton Roads SQL Server User Group. Rumor has it, they’re serving wings!

Rolling Back

I’ve had a number of thoughts rolling around my head lately and they’re somewhat related so here goes.

I’m going to start with a depressing headline and URL: Civilization will end in 2050. Ok, perhaps that’s more of an alarmist headline than anything else. I posted this elsewhere, and some folks correctly pointed out that this could have been written by Thomas Malthus over two centuries ago.  And it’s true; prognosticators have predicted the fall of humans since probably before recorded history.  How many of us recall in the last century alone predictions of world-wide famine, a world-wide ice age, comets smashing into us and other horrible events.

Some were prevented, such with the Green Revolution. Some were corrected with better data or a better understanding of the data. Some where just… nuts.  That said, some predictions can be made with 100% certainty, even if the bounds on the actual circumstances are fuzzy. I’ve been giving this particular nugget a lot of thought.  Since I’ve passed that magical (at least to humans) mark of a half-century, I’ve given more thought to what I call my expiration date. It’s a fact. At some point I will cease all biological functions and will be dead.  I can’t escape that. Every day the odds increase a very tiny percentage.  So, I am predicting my own death, most likely sometime in the next 1/2 century. There’s a very slight possibility it could be tomorrow, and a much greater chance it could be 40 years from now, and based on current medicine an absolute certainty 80 years from now.  You’ll note I hedged that last one. It’s quite possible in the next few decades we figure out how to extend the human lifespan by decades if not centuries.  But I’m not counting on it. And even then, there are no guarantees I’m not hit by a driver while biking or some other catastrophic event.

That said, right now I’d say the odds are decent I’ll be alive in 2050. And almost certainly my kids will be (and very likely I’ll have grandkids by then).  So, am I worried that the world will end in 2050? Yes and no. For one thing, there’s time between now and then for a lot to happen. But, as we get closer, it’s going to be harder and harder to forestall the impacts of higher levels of CO2 in the atmosphere and oceans. Yes, to be clear, I believe the evidence supports that humans are greatly impacting the climate via CO2 and other emissions.

But I’m also optimistic that we’ll work harder, especially as things get worse, to stave off the most pessimistic scenarios. But it won’t be easy and the longer we wait, the harder it will be.

While anecdotal, I’ve already seen the impacts in my life, earlier springs, caves that once held ice year round no longer do, less snow in the winters, etc. It worries me. But I also still have some hope.

That said, an analogy that’s been rolling around in my head that isn’t 100% perfect but fits me as a DBA.

Many of us in the SQL server community have started a long running transaction, only to realize we’ve brought the server to a halt. In other words, no other user can access their data until our transaction is complete. We can abort the transaction, but if we wait too long, that can be a far worse solution than trying to stop it early on. For the non geeks, in other words, if our transaction is going to update 1 billion records and we realize 5 minutes in that it will take 10 hours, we can abort it and it should take about 5 minutes to rollback to the original state. This means we’ve only brought the server to a halt for about 10 minutes. However, if we ignore the problem, and keep pretending it’ll go away and we wait say 9 hours and then finally decide, “oh what the heck, let’s try to fix the problem now” it might take another 9 hours to rollback. In other words, by waiting to long to resolve the problem, instead of being 10 minutes, it can be 1080 minutes.

In other words, the longer you wait, the harder it can be to recover. I think that’s where we’ll be at by 2050. Whether or not I’m still alive (though I plan to be!)


We’re all experts on everything. Don’t think so? Go to any middle school or high school soccer game and you’ll be amazed at how many parents are suddenly experts on soccer. It’s also amazing at how many parents are parents of future NCAA Division I scholarship soccer players.

Seriously though, we’re all guilty of this from time to time. I’ve done it and if you’re honest, you’ll admit you’ve done it.

Yesterday the world suffered a loss, the near destruction of Notre Dame.  Early during the fire our President tweeted:

“Perhaps flying water tankers could be used to put it out. Must act quickly!”

As many have pointed out, this was actually a terrible idea. The idea of dropping 100s of kilograms of water onto an already collapsing roof is most likely to do more damage than not. But, while I think it’s easy to mock the President for his tweet, I won’t. In some ways it reminds me of the various suggestions that were made last summer during the Thai Cave Rescue. We all want to help and often will blurt out the first idea that comes to mind.  I think it’s human nature to want to help.

But, here’s the thing: there really are experts in the field (or to use a term I see in my industry that I dislike at times: SME (it just sounds bad) Subject Matter Expert.)

And sometimes, being a SME does allow you to have some knowledge into other domains and you can give some useful insight. But, one thing I’ve found is that no matter how much I know on any subject, there’s probably someone who knows more. I’ve written about plane crashes and believe I have a more than passing familiarity in the area. Perhaps a lot more than the average person. But, there’s still a lot I don’t know and if I were asked to comment by a news organization on a recent plane crash, I’d probably demur to people with far more experience than I have.

Having done construction (from concrete work in basements to putting the cap of a roof on), I again, have more than a passing familiarity with construction techniques and how fire can have an impact. That said, I’ll leave the real building and fire fighting techniques to the experts.

And I will add another note: even experts can disagree at times. Whether it’s attending a SQL Saturday or the PASS Conference itself, or sitting in a room with my fellow cave rescue instructors, it can be quite enlightening to see the different takes people will have on a particular question. Often no one is wrong, but they bring different knowledge to the table or different experiences.

And finally, you know what, sometimes the non-expert CAN see the problem, or a solution in a way that an expert can’t. But that said, at the end of the day, I’ll tend to trust the experts.

And that’s the truth because I’m an expert on punditry.

Followers and CPR/First Aid

Yesterday, I performed a little social experiment and was pleased to find it worked. I’ve got to say, sometimes it’s the small things that make me happy.

Despite the below zero (Fahrenheit, so really cold, not that warm-cold of 0°C) temperatures, my son and I decided to head up to a local state park and do a hike.  Surprisingly, OK, maybe not, when we arrived, the parking lot was completely empty.  It had been plowed, but there was still a layer of snow over the entire thing, so it was impossible to see where the parking lines were. Now in the summer, this parking lot can be completely full, but I wasn’t too worried about that occurring when the temp was about -4°F.

So, which way to park? Well, there was some sun, so I figured I’d park so that the windshield would get the most sun and hopefully warm up the car just a bit while we hiked. I was sure at the time and later confirmed, this was at a 90° angle to the way the parking lines run. Ironically it was also about 90° colder than the summer temps!

Even when we started hiking, no one else had shown up. But, I have to admit, in the back of my mind I had to wonder if I would start a trend.

Sure enough, 1.5 hours later, when we arrived back at the car there were 3 other cars.  Not only were they parked in the same orientation, they were all parked right next to my car.  This parking lot probably covers 3 acres. They could have parked pretty much any place they wanted in any direction they wanted. But, because I had randomly picked a spot (and not so randomly a direction) 1 car was parked next to me in the same orientation and the other 2 parked facing us.

So what does this little experiment have to do with First Aid or CPR? Have you ever been at an event when someone has a medical event and at first no one reacts? It’s actually fairly common.  Everyone is standing around waiting for someone else to react. But once someone reacts, others tend to follow.  Be that person that others follow.  Learn CPR and learn First Aid so that when something happens, you can be the first to react. Sometimes people just need a leader to follow; and often they don’t necessarily realize it.

There’s no good reason anyone else parked just like I did, and yet they did. But there is a good reason for people to follow you if you can be the first to react in an emergency.  And you don’t have to be an expert. Obviously it didn’t take “expertise” to park yesterday, but people followed anyway. You don’t have to be an EMT or paramedic to react at a medical emergency. You can be the person that simply shouts, “Call 911” and gets people reacting.

That said, I still highly recommend taking a CPR and First Aid course. Not only do you learn very useful medical response skills, it will help you be that person that reacts first.

And stay warm!