Monthly Archives: January 2026

Tough Times

Posted on by
1

There’s many reasons I maintain this blog. It started talking about my thoughts on design (both database and real word) and on metacognition and other topics. Often I spoke about caving and the NCRC. But sometimes I write, because I have to. This is one of those.

Let me start with two recent things shaping my current thought processes. My Pharm exam this morning. I won’t get a grade until Friday probably. And despite how hard I studied for it, I don’t expect it to be good. Pharmacology is my nemesis. It stresses me out. So, I’m completely stressed right now and to be honest, wondering if all the stress is worth it. But that’s a topic for another day. (Though you can read my thoughts from the end of last semester here.)

The second part was learning one of our cats has cancer that has metastasized. Many folks don’t believe us when we say we have two cats because they never see this one. Pisantar definitely is a bit skittish and tends to hide when company is around. But, of the two, he’s ultimately the more curious and probably more intelligent one. I have bonds with both cats, but sometimes I think I identify with Pi (as we call him) a bit more. So, that double whammy has me down.

But, what I really wanted to write about is something that finally gelled in my mind the other night. By now we’re all familiar with the shocking killing of Alex Pretti. When I saw the first video released I was shocked, upset, and sick to my stomach. Things haven’t gotten much better. If anything in some ways worse. And then the other night it hit me. He was an ICU nurse. He was one of us.

In over thirty-five years of IT, I’ve worked with teams large and small. And along the way, a few have passed, all from natural causes, including Covid. Honestly, one, given his health, didn’t surprise me at all. But, I’ll be honest, even though I’ve made friends, the closeness has never been as much as it has been with my coworkers in the ER. Even ones I might not consider close friends, I share a close, intimate bond with. I think it’s because in my IT jobs, the worst that could happen was a database might crash, some money might be lost, even jobs might be lost, but no life was saved or lost. Obviously in the ER it’s different. We have a common goal and a common enemy. We struggle to keep people alive for one more day. It doesn’t matter who they are or why they are there. They need help. We help.

In the ER I’ve encountered the best of the people and the worst. I’ve been punched. I’ve seen my coworkers be called the worst names (I once threatened to have a person ejected because of their behavior). I’ve seen threats be made. But I’ve also seen the family member cry on the shoulder of a nurse because we saved their mother. I’ve seen the wife smile, knowing her husband’s chest pains are just indigestion from her dinner, not a heart attack that could have made her a widow. I’ve seen the satisfaction on the team’s face when our compressions and meds were successful and we know the person was discharged, neurologically intact. We’re there. We’re making a difference, no matter who the patient is.

And, no matter who our coworkers are. There are coworkers whose political believes I disagree with. There are the coworkers who have rubbed me the wrong way. But, when push comes to shove, those are the very same coworkers I know will do everything in their power to try to save someone. We work as a team. We are a team.

No one I know goes into Emergency Medicine for the money. We do it because it’s who we are. Because we want to make a difference. We want to be part of something bigger and better than our individual contributions. We want to be part of a team.

Now in some ways, the ICU is a different place. It’s quieter. Far less chaotic. But at the end of the day, it’s the same thing. People doing their best to help their patients. People are there to make a difference. They’re a team.

And this extends beyond the ER. Many of my coworkers are also EMTs and paramedics. Or rescue animals. Or do other acts of service. It’s why I’ve done the NCRC for so long, it makes a difference. We’re one.

So, I realized, when Alex Pretti died, it was like a coworker died. It was someone I could have been close to. Someone I could have worked with to save a life.

I couldn’t imagine going into work knowing one of my coworkers had had their last shift. That one of my coworkers had run their last code. That one of my coworkers had pulled drugs from the Pyxis for the last time. We had lost one of our own. When I saw his flag draped coffin rolling out for the last time with his coworkers standing there, I realized, I was there too, in spirit.

It could have been any of the team I work with. And I realized, too, that knowing me and my spirit and desire to be out there, helping, it could have been me.

T-SQL Tuesday #194 – Why I Don’t Take CRaP from Anyone

Posted on by
1

It’s been quite a while since I participated in a T-SQL Tuesday and with my new career path, less reason to do so, but this topic appealed to me.

This month Louis Davidson asked us to talk about a mistake we made and what we learned from it.

I have two mistakes. The first I call the White Ford Taurus Mistake. Back before the turn of the millennia (yes, I’ve been working with SQL Server for that long) I was consulting at a start-up (where I was later hired as their IT manager) that helped car dealers sell their inventory online. We were young, fast, and nimble. And sometimes had no clue what we were doing. I technically didn’t make this mistake, but I was there when it happened. A dealer had uploaded their inventory but there was a mistake for one vehicle. So we had a request to update the details on that car. So an update was made on the production server. I forget the exact statement but it was something like “Update AUTO set MANUFACTURER=’Ford’, Type=’Taurus’, Color=’White'” You’ll notice the distinct lack of a WHERE clause. Since we did this without any sort of explicit transaction around it, suddenly every car in the database was now a White Ford Taurus.

I can assure you we didn’t make that level of error again!

The second error was in some ways worse and led to the title of this blog. This occurred two years later when we had matured, had actual servers and had moved into a datacenter in New York City (incidentally it’s now the Google building at 111 8th Ave, but back then had several different hosting providers). I had been taken a lot of trips to our datacenter over the past few months so my boss suggested for this trip I take my wife and we catch a show. It was a great plan. Until it wasn’t. This trip was supposed to be fairly simple:

  • Failover our Classified Ads database to our backup server – This was basically telling Enterprise Manager to backup and restore the database from production to the backup server. (I’ll be honest, I forgot what this feature was called, it was so long ago).
  • Update a hardware driver on the main server
  • Failback

In theory this should have taken about 1-2 hours tops.

The first time I tried the failover it failed. I forget the exact reason, but it was a simple fix. However, what I didn’t realize was that Enterprise Manager had flipped the direction of the migration on the screen.

So after fixing the initial problem I hit the button again. This time it was successful. And very fast. Too fast. It took me a second to realize the problem. Enterprise Manager had done exactly what I told it to do. In this case it had copied an empty database over the production database.

No problem, right, simply restore the most recent backup.

Big problem. There was no recent backup.

This was one of the worst calls I ever had to make to my boss. Fortunately he took it in stride, called our Wisconsin office and had them start reloading the data. Since the only product involved was our classified ads database, the data had a high churn rate meaning over the course of 7 days it was all new data. So they basically had to do 7 days of loads in one day. Unfortunately this was going to take about 8 hours and meant that I had to postpone any plans until then. Once done, I headed back to the data center, made a backup, then did the failover correctly, and tried the driver update. This ran into its own issues, but isn’t the subject of this post.

I ended up leaving the datacenter at about 3:00 AM. Needless to say my wife and I didn’t have our date night.

So what did I learn from this? Reviewing and planning any production changes. As a result I developed what we internally called our Change, Review, Analysis, Plan document, or CRAP for short, but since that didn’t look good in an email, we dropped the Analysis and it simply became the Change, Review, and Plan document.

Somewhere I probably still have a copy of one, but the basics were a stratification of the risk and steps taken to reduce it. For example, in the above mistake, a key step would have simply been “ensure recent backup”. Had I done that before my mistake above, the recovery would have been about 20 minutes, not 8 hours.

So the goal of the document wasn’t just to prevent mistakes, but to assume they would happen and to analyze the impact and what steps could be taken to minimize the risks and reduce the recovery time if necessary. It also had an area for determining who needed to sign off on a change. Something that might have a minimal impact (such as adding a column to a table) might only require my signature and one other. On the other hand, if say we were doing a full fail-over test where if things went bad we could have a complete outage for an extended time, that would require the CEO to sign off on.

Over the years, sometimes I had employees of mine grumble when I’d reject the CRaP they had submitted to me for not being detailed enough or covering all the bases. But, in at least one case, one of my network managers came back after such a rejection and said, “Hmm, you’re right. I missed this particular failure mode. I’m going to do this in person instead of remotely.” The update he was doing ran smoothly, but if it had failed, the outage would have been measured in minutes not hours.

So sometimes, assume you’ll make mistakes, but have a plan for handling them.

2026 A Year in Preview

Posted on by
Reply

For several years now I’ve set some goals for the upcoming year. These are not resolutions per se.

Last year I didn’t set too many goals because I knew I’d be starting PA school. This year my list is perhaps even shorter. But here goes.

  • Finish my didactic year of PA school. I’m two semesters down, one semester to go. It won’t be easy, but I think I’ve got it.
  • Start my clinical year. I have a total of 10 four-week clinicals I have to do. They’ll extend until April next year. This will be difficult and involve a lot of travel, but I think I’ve got it. But ideally at least one or two will be local to Albany.
    • Related to this one: Going to South Africa. I applied for both an international rotation in South Africa (with a focus on procedures) and a small scholarship. I received both. The scholarship doesn’t come close to covering the additional cost of the rotation, but it helps take some of the sting away.
  • Last year I mentioned seeing friends. While always a goal, I don’t think I’l have too much time for that.
  • Biking – I’ll definitely continue doing this because I need the physical activity and it helps my mental health.
  • Beyond that, not many goals this year. Only a few goals, but they’re big ones.