Let me Email that!

This week’s post is a short one, but hopefully a practical and useful one. I had a recent need to send an email to about 3 dozen folks. Prior to this particular email, I had been updating them with general information and was simply using the BCC function of my email client to send out the email. But in this particular case I needed to email them login information specific to them. I didn’t relish the idea of crafting 3 dozen separate emails and making sure I got the correct information into each. So, I turned to the tool closest at hand: PowerShell and in about 3 minutes of writing, and 10 more minutes of debugging and tweaking, had something that worked great.

param([parameter(mandatory)] $course_id, [parameter(mandatory)] $course_description)

$course_info = invoke-sqlcmd -server "sql_server" -query "select ct.Course_Type_Name, c.Location, c.City,c.State from Courses c
inner join course_types ct on ct.Course_Type_ID=c.Course_Type_ID
where course_id=$course_id" 


$class_list=invoke-sqlcmd -server 'sql_server' -query "select p.persno, first_name,LAST_NAME, email1 from courses_taken ct
inner join people p on p.PERSNO=ct.PERSNO
and ct.Course_ID=$course_id"

foreach ($person in $class_list)
{
    $body = "Thank you for participating in the recent $($course_info.Course_type_name) $course_description course at $($course_info.location) in $($course_info.City), $($course_info.state). Your class information has been entered into the Generic Class database. 
If you wish to review your unofficial transcript, you may login in at https://www.example.com/Account/Login.aspx. 
If you have not created an account before you can create an account at https://www.example.com/Account/Register.aspx. 
Please be sure to use the following email:$($person.email1) (you can update it after creating your account) and your Generic Class ID: $($person.persno). Please note this is NOT your OTT ID. 
If you have issues or questions, please contact Joe Smith at mailto:jsmith@example.com. Thank you and we look forward to seeing you at future training!"

    Send-MailMessage -From "jsmith@example.com" -SmtpServer Mail_Server -Subject "Generic Class: Your Generic Class Login and unofficial transcript" -Body $body -to $person.email1
}

Every course in this particular database has an ID. And a course can have a specific course information. The first query gathers that information.

The second query then gets the personal information for each student who took that particular course and puts it into a class_list object.

I then simply iterate over the object and send an email with the personalized details to each student.

It literally took me more time to tweak the text and formatting of the body of the message than it did to write the rest of the script.

I suppose I could have figured out my email client’s mail merge feature, or done something in Word which I know also has such a feature but that would have taken far longer. I also could have spent more time making things more generic (perhaps passing in the body and having it automatically formatted, or sending as a nice HTML message with appropriate formatting. But sometimes, quick and dirty is the way to go. And now I have a script I can customize for future use.

Not bad for about 15 minutes worth of work.

Coping

I’m going to be a bit more open than I usually am in my blog posts. I think it’s time for a bit of transparency.

Let me start by saying that overall, despite the impact of Covid, the last 15 or so months have not been terrible for me. Far from it. In fact I’ve been very fortunate. So this post isn’t a rant or a series of complaints. It’s really a short reflection.

Last year for example, I biked more than I had in years, and in fact did my first century ride since college. I presented at PASS Summit for the first time, albeit virtually. I got to spend more time with my kids. Fortunately, no one close to me had a serious case of COVID nor died from it (though I had friends and former coworkers who did lose people close to them).

I even managed to organize and pull off a cave rescue training class late last summer. And of course just pulled off another weeklong course this month, and will help with another one in late August.

So overall, it’s been a pretty decent 15 months.

But lately I’ve noticed things aren’t necessarily where I want them to be. My motivation levels have been off. I’ve got at least 3-4 ideas for more articles for Redgate. But, I find myself finding reasons to put off writing them. I’ve got a few other projects that I haven’t made progress on. I need to finish tiling a backsplash in my bathroom, patch a hole in the wall in the downstairs bathroom from when I put in a fan, and a few more.

But honestly, the idea of launching into such projects just makes me go “bleah”.

I think too some of the frustration in my inability to attract new clients like I was hoping to this year has put me into the “bleah” mode, and of course in a vicious cycle caused me to put less effort into attracting new clients.

But, ultimately, I’m writing this post not because I’m looking for sympathy or for comfort, but ironically for the exact opposite reason. I find that often people hide the state of their emotional well-being and put on a happy fa├žade, especially on social media, and as a result everyone goes around thinking that everyone else is doing better than they are themselves doing. So, I’m saying, “hey, I’m doing great, but you know what, there are days when life is ‘bleah’ and it’s ok. And if you’re having such days, or even worse, you’re not alone.”

Postscript: I want to add, if you haven’t, check out Steve Jones blog, he’s been daily posting a bunch of coping suggestions. I don’t read them every day, and I suspect Steve would agree with me that take what works for you and ignore the ones that don’t is the way to go. In part his posts helped inspire this one.

“We’re up to plan F”

I managed to skip two weeks of writing, which is unusual for me, but I was busy with other business, primarily last week leading an NCRC weeklong class of cave rescue for Level 1 students. I had previously lead such a class over three weekends last year, and have helped teach the Level 2 class multiple times. Originally this past week was supposed to be our National weeklong class, but back in February we had agreed to postpone it due to the unknown status of the ongoing Covid pandemic. However, due to a huge demand and the success of vaccinations, we decided to do a “Regional” Class just limited to Level 1 students. This would help handle the pent up demand, create students for the Level 2 class that would be at National, and to do sort of a test run of our facilities before the much larger National.

There’s an old saying that no plan survives the first contact with the enemy. In cave rescue this is particularly true. It also appears to be true in cave rescue training classes!

The first hitch was the drive up the the camp we were using. The road had been stripped down to the base dirt level and they were doing construction. Not a huge issue, just a dusty one. But for cavers, dust is just mud without the water. But this would come into play later in the week.

Once at the camp, as I was settling in and confirming the facilities, the first thing I noticed was that the scissors lift we had used to rig ropes in the gym last time was gone. A few texts and I learned it had only been on loan to the camp the past two years and was no longer available. This presented our first real challenge. How to get ropes up over the beams 20-30′ in the air.

But shortly after I realized I had a far greater issue. The custom made rigging plates we use to tie off the end of the ropes to the posts were still sitting in my garage at home. I had completely forgotten them. This was resolved by a well timed call to an instructor heading towards the camp, who via a longer detour then he expected, was able to get them. Fortunately, had that call waited another 5 minutes, his detour would have probably doubled. So the timing was decent.

I figured the week was off to a good start at that point! Honestly though, we solved the problems and moved on. I went to bed fairly relaxed.

All went well until Monday. This was the day we were supposed to do activities on the cliffs. Several weeks ago, my son and I, along with two others had gone to the cliffs, which were on the same property as the camp, but accessible only by leaving the camp and accessing from a public road, in order to clear away debris and do other work to make them usable. I was excited to show them off. Unfortunately, due to the weather forecast of impending thunderstorms all day we made the decision to revise our schedule and move cliff day to the next day. There went Plan A. Plan B became “go the next day.”

On Tuesday I and a couple of other instructors got in my car to head to the cliffs in advance of the students so we could scope things out and plan the activities. We literally got to the bottom of the road from the main entrance to the camp where we were going to turn on to the road under construction, only to find a the road closed there with a gaping ditch dug across it. So much for Plan B. We went back to the camp, told students to hang on and then I headed out again, hoping to basically take a loop around and approach the access road to the cliffs from the opposite direction. After about a 3 mile detour we came to the other end of the road and found it closed there. Despite trying to sweet talk the flag person, we couldn’t get past (we could have lied and said we lived on the road, but after 8-10 other cars would have arrived in a caravan saying the same thing we thought that might be suspicious). There went Plan C. We called an instructor back at the camp and headed back.

We got there and turns out an instructor had already come up with Plan D, which was to see if we could access the cliffs by crossing a field the camp owned and going through the woods. It might involve some hiking, but it might be doable. While there are dirt-bike paths, there’s nothing there that worked for us. So that plan fell apart. We were up to Plan E now. Plan E was proposed to further swap some training, but we realized that would impact our schedule too much. Now on to Plan F. For Plan F, we decided to head to a local cave which we thought would have some suitable cliffs outside.

That worked. It would out quite well actually. We lost maybe an hour to 90 minutes with all the plans, but we ultimately came upon a plan that worked. We were able to teach the skills we wanted and accomplish our educational objectives.

Often we wake up with a plan in our heads for what we will do that day. Most days those plans work out. But, then there are the days where we have to adapt. Things go sideways. Something breaks, or something doesn’t go as planned. In the NCRC we have an unofficial motto, Semper Gumby – “Always be Flexible”. Sometimes you have to completely change plans (cancelling due to the threat of thunderstorms), others you may have to try to adapt (finding other possible routes to the cliffs) and finally you may need to reconsider how to meet your objectives in a new way (finding different cliffs).

My advice, don’t lock yourself into only one solution. It’s a recipe for failure.

“Man down!”

Last week I wrote about how in many crisis situations you should actually stop and take 5 minutes to assess the situation, take a deep breath, and maybe even make a cup of tea. The point was, in many cases, we’re not talking life or death, and by taking a bit longer to respond we can have a better response.

I pointed out that you don’t always have that luxury. That happened to my mom’s partner within days of me writing last week’s post. While at work at a local supermarket chain, he heard someone shout “Man down”. Next thing he knew, a young man was laying on the floor having seizures. He jumped into action and provided the appropriate, immediate first aid. This included telling someone to call 911. Apparently no one else, including his manager responded at first. But, he had learned how to respond in his basic training in the Army decades ago. That training stuck.

My mom called me to talk about this and wondered why no one else had responded (she knows of my interest in emergency response and the like). I pointed out it’s a variety of factors, but often comes down to people don’t know how to respond, or they’ve assumed someone else has already responded. This discussion prompted a quick Facebook post by me that that I’m expanding upon here.

Let me ask you this, if someone collapsed in front of you at the mall, would you know what to do? What would you do? Would you do it?

The reality is, unfortunately many would not respond. So here’s my advice.

Get some training

You do not need to become an EMT to respond. In fact most training can be done in just a few hours.

Take a First Aid and a CPR course. Make sure the CPR course includes a segment on how to use an AED (Automatic External Defibrillator). I’ve taken several such courses over the years and try to remain certified.

Take a Stop the Bleed class. This is a bit different from your standard First Aid class. I haven’t taken it yet, but plan to when I can find one near me (I may even look into getting one setup when I have a bit more free time).

“911, what’s your emergency?”

Call 911. Anyone can do this. I would recommend even teaching even your young children to do this if they find you or someone else unconscious. Even if they can’t communicate much details, 911 operators are trained to gather what information they can, have ways (usually electronically) of determining the address and dispatching help. (Please note, if your child or someone else calls 911 by accident, please do NOT hang up. Simply let them know it was a mistake. It happens, they understand. But if they aren’t made aware, they WILL dispatch resources).

TELL someone specific to call 911. If you’re about to render aid, do NOT assume someone has already called 911 or will. In a crowd, groupthink happens and everyone starts to freeze and/or assume someone else has it handled. My advice, don’t just say “someone call 911”. Point to a specific person and tell them to call 911. Odds are, they will do it. In many cases in an emergency, folks are simply looking for someone to take charge and to give them direction. Now, someone else may have already called 911, or it may end up being multiple people will be calling 911. THAT IS OK. That’s far better than no one calling if it’s an emergency. In the event of a heart attack minutes count. This means that the sooner 911 is called, the better.

Respond

This may sound obvious, but be prepared to act. Again, it’s a common trope that in large crowds, people tend NOT to act, because in part they expect someone else already has it covered. Be that person who does act.

Years ago in the northern Virginia area, I witnessed a car get t-boned on the far side of an intersection from me. There were 3 lanes of traffic in either direction. NO ONE stopped to check on the drivers. I had to wait for the light to change before I could cross the intersection and check on them. Fortunately, the driver of the car I checked on was fine, other than some very minor injuries from their air bag deploying. And by this time, another witness had finally stopped to check on the 2nd car. They too were fine. But several dozen people had witnessed the accident and only the two of us had responded. If they drivers had been seriously injured and no one had responded, things could have been much worse for them.

Carry gloves, maybe more

Carry nitrile gloves with you. Sounds perhaps a bit silly or trite, but they don’t take up room and you can toss them in your backpack, glove compartment (yes, really you can put gloves in there), your purse etc. If you do come across someone who is injured, especially if blood or other bodily fluids are present, don them. I even carry a tiny disposable rebreather mask for CRP in my work backpack. Takes up no room but it’s there if I need it.

When you enter public buildings, look to see if they have a sign about AED availability. Note it and if possible where it is. In addition to telling someone to call 911, be prepared to tell someone “Get the AED, I think there’s on next to the desk in reception.”

Get your employer involved

Get your work to sponsor training. And honestly, while many companies might offer video tutorials with a quick online quiz at the end, I think they’re a bare minimum. I think hands on training is FAR more effective. There’s a number of reasons for this as I understand it, including the fact that you’re often engaging multiple pathways to the brain (tactile as well as visual and auditory) and a certain level of stress can actually improve memorization.

Seeing a video about how to use an AED is very different from holding a training unit in your hands and feeling its weight and hearing it give you instructions directly. Applying a bandage is far more realistic when your mock patient is laying there groaning in pain. Even getting into the action of telling someone “Call 911” is far more impactful when you do it in a hands-on manner and not simply checking a box in an on-line quiz.

Find out what resources are available in the office. Is there a first aid kit? What’s in it? For larger offices, I would argue they should have an AED and perhaps a Stop the Bleed kit. When’s the last time the AED batteries were tested? Who is responsible for that?

This works

In the case of the “man down” that prompted this post, they are reportedly doing fine and suffered no injuries.

I know of a local case, at a school where a student collapsed. A coach and the school nurse responded. And while the nurse especially had more training, what saved the students life was having an AED on site and available. Even if the school nurse or coach had not been there, in theory any bystander could have responded in a similar fashion.

As I said above, you don’t have to be a highly trained EMT or the like to make an impact and save someone from further injury or even save a life. You simply need to have some basic training and be willing to respond.

Take 5 Minutes

This weekend I had the pleasure of moderating Brandon Leach‘s session at Data Saturday Southwest. The topic was “A DBA’s Guide to the Proper Handling of Corruption”. There were some great takeaways and if you get a chance, I recommend you catch it the next time he presents it.

But there was one thing that stood out that he mentioned that I wanted to write about: taking 5 minutes in an emergency. The idea is that sometimes the best thing you can do in an emergency is take 5 minutes. Doing this can save a lot of time and effort down the road.

Now, obviously, there are times when you can’t take 5 minutes. If you’re in an airplane and you lose both engines on takeoff while departing La Guardia, you don’t have 5 minutes. If your office is on fire, I would not suggest taking 5 minutes before deciding to leave the building. But other than the immediate life-threatening emergencies, I’m a huge fan of taking 5 minutes. Or as I’ve put it, “make yourself a cup of tea.” (note I don’t drink tea!) Or have a cookie!

Years ago, when the web was young (and I was younger) I wrote sort of a first-aid quiz web-page. Nothing fancy or formal, just a bunch of questions with hyperlinks to the bottom. It was self-graded. I don’t recall the exact wording of one of the questions but it was something along the lines of “You’re hiking and someone stumbles and breaks their leg, how long should you wait before you run off to get help.” The answer was basically “after you make some tea.”

This came about after hearing a talk from Dr. Frank Hubbell, the founder of SOLO talk about an incident in the White Mountains of New Hampshire where the leader of a Boy Scout troop passed out during breakfast. Immediately two scouts started to run down the trail to get help. While doing so, one slipped and fell off a bridge and broke his leg. Turns out the leader simply had passed out from low blood sugar and once he woke up and had some breakfast was fine. The pour scout with the broken leg though wasn’t quite so fine. If they had waited 5 minutes, the outcome would have been different.

The above is an example of what some call “Go Fever”. Our adrenaline starts pumping and we feel like we have to do something. Sitting still can feel very unnatural. This can happen even when we know rationally it’s NOT an emergency. Years ago during a mock cave rescue training exercise, a student was so pumped up that he started to back up and ran his car into another student’s motorcycle. There was zero reason to rush, and yet he had let go fever hit him.

Taking the extra 5 minutes has a number of benefits. It gives you the opportunity to catch your breath and organize the thoughts in your head. It gives you time to collect more data. It also sometimes gives the situation itself time to resolve.

But, and Brandon touched upon this a bit, and I’ve talked about it in my own talk “Who’s Flying the Plane”, often for this, you need strong support from management. Management obviously wants problems fixed, as quickly as possible. This often means management puts pressure on us IT folks to jump into action. This can lead to bad outcomes. I once had a manager who told my team (without me realizing it at the time) to reboot a SQL Server because it was acting very slowly. This was while I was in the middle of remotely trying to diagnosis it. Not only did this not solve the problem, it made things worse because a rebooting server is exactly 100% not responsive, but even when it comes up, it has to load a lot of pages into cache and will have a slow response after reboot. And in this case, as I was pretty sure would happen, the reboot didn’t solve the problem (we were hitting a flaw in our code that was resulting in huge table scans). While non-fatal, taking an extra 5 minutes would have eliminated that outage and gotten us that much closer to solving the problem.

Brandon also gave a great example of a corrupted index and how easy it can be to solve. If your boss is pressuring you for a solution NOW and you don’t have the opportunity to take those 5 minutes, you might make a poor decision that leads to a larger issue.

My take away for today is three fold:

  1. Be prepared to take 5 minutes in an emergency
  2. Take 5 minutes today, to talk to your manager about taking 5 minutes in an emergency. Let them know NOW that you plan on taking those 5 minutes to calm down, regroup, maybe discuss with others what’s going on and THEN you will respond. This isn’t you being a slacker or ignoring the impact on the business, but you being proactive to ensure you don’t make a hasty decision that has a larger impact. It’s far easier to have this conversation today, than in the middle of a crisis.
  3. If you’re a manager, tell your reports, that you expect them to take 5 minutes in an emergency.

Changing Technologies – T-SQL Tuesday

Select <columns> from Some_Table where Condition=’Some Value’

T-SQL Tuesday Topic

The above statement is pretty much the basis of what started my current career. Of course it actually goes back further than that. I have a Computer Science Degree from RPI. So I’ve done programming, learned hardware and more. I even took an Intro to Databases course while at RPI. I still recall the professor talking about IBM and something called Structured Query Language. The book had a line that went something like “while not the most popular database technology, its use may grow in the future.” Boy did it.

When I first started working with SQL Server, it was 4.21 and for a startup. I had a lot to learn. Back then, a lot was by experience. Sometimes I made mistakes. But I learned.

When I started at that startup, if one could write basic queries and backup and restore a database, one was a half-way decent DBA. Knowing how to tune indices was a definite bonus, as was knowing things like how to set up log-shipping and replication.

Back then, besides experience, I learned new stuff two ways: SQL Server Magazine and the SQL Connections conference. Work paid for both. It was worth it. But honestly, there wasn’t too much to learn. But there also weren’t as nearly as many resources as there were today.

Fast forward 30+ years and here I’ve written a book, worked for several startups, regularly write about databases and database related topics, and often presented at User Groups, SQL Saturdays and at the now defunct PASS Summit. Today as a consultant I regularly touch the SQL Server Query Engine, SSAS, SSRS, SSIS, use PowerShell, write the occasional C# and VB.Net, sometimes do work on a Linux machine or VM and more. A lot has changed.

Obviously the technology has changed. So how have I responded? By doing what I said above. This may sound like a tautology or even circular reasoning but it’s true. When I would go to a SQL Saturday, I’d often attend 3-5 different topics. I’d learn something. But then I started presenting. And that forced me to learn. As much as I may like to think I know about a topic, when I go to present about it, I like to ensure I know even more. This forces me to read white papers, other articles and perhaps attend other sessions.

When I’ve written an article, I’ve often had to do a lot of research for it.

So strangely, I would say a bit part of keeping my skills up to date is not just learning from others, but from teaching. Teaching forces me to keep my skills up.

In summation, I’ve responded by learning from others, but also forcing myself to teach myself before I taught others. It’s a feedback loop. The more technology changes, the more I reach out and learn and the more learn, the more I do outreach.

The impetus for this week’s blog was Andy Leonard’s call for a T-SQL Tuesday topic.

Hiring

Not sure why it came to mind last night, but I was thinking of the best hire I never made. This expanded into me thinking about folks I have hired over the ages. As a Director of IT and later a VP of IT, I’ve had to make a lot of hires over the years, some better than others. Even when I can’t remember their names (an unfortunate weakness of mine) I can almost always remember their faces and how they worked out. And fortunately, most of them worked out quite well, even the ones who surprisingly might think they didn’t.

Looking back, I would say there was probably only one person I absolutely should not have hired and she was the only person I ended up having to let go because of performance issues. There were a few how were less than stellar, and a few I had to let go because of budget cuts, but even those weren’t necessarily bad hires.

But then there’s the one that “got away” and honestly, when I reflected upon it, I was glad, for both of us. Back in the early days of the first dotcom bubble I was working for a company that was quickly expanding. I can’t recall how many interviews a day I was doing, but it was a lot. We were looking to ramp up quickly and I couldn’t afford to be too picky. That said, some of my best hires came during that period.

In this case she was an ideal candidate, both on resume and in person. She had a great college background, ticked all the checkmarks in terms of classes taken and experience. She did great during the interview, both technically and in terms of how I thought she’d be for the team I was looking to build. In fact, looking back, I think she would have been the first member of said team and as such would have been a good role model for others.

There was only one issue, and we both recognized it in time. We were a startup. We didn’t ask that stereotypical (and I think bad) question of “where do you see yourself in 5 years?” because, heck, we didn’t know where we’d be in 5 years. We didn’t have a clear career path of growth for employees. I mean it was obvious we’d grow and there would be steps up, but there was no clear org chart.

On the other hand, companies like GE, especially back then, had a very clear progression path. If you wanted management, you knew the path to take and it was pretty clear that both parties would work to make it happen.

And, it became apparent, she wanted to know where she would be in 5 years. And there was absolutely nothing wrong with that. We made her the offer, but I half-hoped she’d turn it down and was relieved in some ways that she did. Yes, she would have been a great hire for us. However, honestly, for her own career, it probably would have been a mistake.

But, I have to wonder what things would have been like had she joined the team. She would have been great. She’s the one that got away. And I’m OK with that.

“It’s Just a Simple Change”

How often have we heard those words? Or used them ourselves?

“Oh this is just a simple change, it won’t break a thing.” And then all hell breaks lose.

Yet, we also hear the reverse at times. “This is pretty complex, I’ll be surprised if it works the first time, or if it doesn’t break something.” And yet then nothing bad seems to happen.

We may observe this, but we don’t necessarily stop to think about the why. I’ve seen this happen a lot in IT, but honestly, I’ve seen this happen elsewhere and often when we read about accidents in areas such as caving, this also holds true.

I argue that in this case the perception is often true. Let me put in one caveat. There’s definitely a bias in our memory where we don’t recall all the times where simple things don’t break things, but the times it does, it really stands out.

The truth is, whenever we deal with complex systems, even simple changes aren’t so simple. But we assume they are and then are surprised when they have side effects. “Oh updating that path here won’t break anything. I only call it one place, and I’ll update that.” And you’re good. But what you didn’t realize was another developer liked your script, so made a copy and is using it for their own purposes and now their code breaks because of the new path. So your simple change isn’t so simple.

Contrast that to the complex change. I’m in the middle up refactoring a stored procedure. It’s complex. I suspect it’ll break something in production. But, honestly, it probably won’t. Not because I’m am awesome T-SQL developer, but, because of our paranoia, we’ll be testing this in UAT quite a bit. In other words, our paranoia drives our testing to a higher level.

I think it behooves us to treat even simple changes with more respect than we do and test them.

In the world of caving we use something called SRT – Single Rope Technique. This is the method we use for ascending and descending a rope. When ascending, if you put your gear on wrong at the bottom, generally there’s no real risk other than possible embarrassment. After all, you’re standing on the ground. But obviously a the top, it’s critical to put your equipment on correctly, lest your first step be your last. Similarly, we practice something known as a change-over; changing from ascending to descending, or descending to ascending while on rope. When changing from climbing to descending you want to make sure you do it correctly lest you find yourself descending at 9.8m/s^2. To prevent accidents, we ingrain in students “load and test your descent device before removing your other attachment point.” Basically, while you’re still secured to something at the top, or to your ascending devices if you’re partway up the rope, put your entire weight on your descent device and lower yourself 1-2″. If you succeed, great, then you can detach yourself from whatever you are attached to at the top, or remove your ascending devices. If somehow you’ve screwed something up and the descent device comes off the top or otherwise fails, you’ve got a backup.

Now, I will interject, getting on rope at the top of a pit, or a changeover is something an experienced caver will have done possibly 100s if not 1000s of times. It’s “a simple change”. Yet we still do the test because a single failure can be fatal. And I have in fact seen a person fail to properly test their descent device. And moreover, this wasn’t in a cave, or other dark or cramped space. It was in broad daylight on the edge of the RPI Student Union! This was about as simple as it could get! Fortunately he heard it start to fail and grabbed the concrete railing for dear life. In this particular case a failure most likely would not have been fatal, but would have caused serious injury.

So, despite having gotten on rope 100s of times myself, I ALWAYS test. It’s a simple change. But the test is also simple and there’s no reason to skip it.

The morale of the story, even your simple changes should be tested, lest you find they’re not so simple, or their failures aren’t so minor.

4/20

I was going to start this post by making a crack about getting any cracks about references to 420 out of the way. But then I realized they’re actually apropos of the intent of this post.

Yes, often when we folks think of the numbers 420 the references to marijuana jump out. Not a habit I’ve ever had any interest in, but I’ve been around it enough to feel its effects and I guess I can understand why others might partake. Growing up in the 70s and 80s I was routinely offered it but always declined due to lack of interest. That said, one thing that I never really dwelt on much was what would happen if I got caught with it. My skin color mattered.

Three events though shaped 4/20/21 for me.

I happened to reread (I had come across it earlier) a post by Eva Kor on Quora. Eva Kor was a twin who survived Josef Mengele’s atrocities and spent much of her life talking about them. She was a living witness to the history of the Holocaust, an event we must never forget. Sadly she is gone now, but her writings and voice live on.

4/20 also happens to be the birthday of George Takei. I recall growing up watching him in reruns of the original Star Trek, playing originally a physicist on the Enterprise, but really best known as the ship’s navigator. To quote Spock Sulu “is at heart a swashbuckler out of the 18th century”. But I later learned he was also instrumental in bringing attention to a dark period of our own US history during WWII, the internment of US citizens of Japanese heritage. He is, at this writing, still a living witness to those dark days. But, the truth is unfortunately, time will eventually silence his great voice. But that does not mean we can be allowed to forget what the US did to its own citizens.

And finally of course 4/20/21 was the reading of the verdict of in the George Floyd murder case. Guilty on all three counts. George Floyd’s life was sadly ended with the words “I can’t breath.” He can’t speak for himself. But fortunately, due to cell phone cameras, and the work of the prosecution, the jury could speak for accountability and hold his murderer responsible.

While the murderer will be held accountable, it will not change the tragedy that such an event should never have happened. There are those that will still argue, “well if he hadn’t resisted arrest…” ignoring the idea that perhaps the initial response while legal, probably should have been handled very differently. Dr. Mengele’s atrocities were considered legal, but that didn’t make them right by any moral compass I am comfortable with. The Supreme Court in Korematsu v. United States held that the government could force Korematsu to be detained because of his heritage. In the case of George Floyd, the defense argued a reasonable officer would do what George Floyd’s murderer did. The jury rejected that argument. Thankfully. But we know all to often where that argument did hold sway. And, honestly will again.

So back to 420. The decriminalization of marijuana is quickly becoming the norm. Even my US Senator Chuck “I never found a camera I didn’t like” Schumer posted on Facebook positively about 420 day. These are steps forward. But, there is still an ugly racial history to the handling and prosecution of crimes related to marijuana in this country. Blacks for example are about twice as likely to be arrested for possession, despite their rate of use being about the same as whites. Like many aspects of the law, it’s clear it’s applied disproportionality and in a huge part based on the color of ones skin. Hence why I never really worried too much about it.

Fortunately here in New York, part of the rollback of marijuana laws is including vacating 10s of thousands or prior convictions and expunging them from individual records (there are some caveats however.) This is a step towards restorative justice.

So 4/20 represents a confluence of events and perhaps a step forward. But despite Eva Kor’s testimonies, George Takei’s work, still going on today, and the conviction of George Floyd’s murderer, we have a long ways to go towards the living up to our ideals. They are the voices calling us to do better. And we must. And we must never think the work is done.

Your Boss Doesn’t Care About Backups!

It’s true. Even if they don’t realize it. Or even if they claim they do. They really don’t.

I’ve made this point before. Of course this is hyperbole. But a recent post by Taryn Pratt reminded me of this. I would highly recommend you go read Taryn’s post. Seriously. Do it. It’s great. It’s better than my post. It actually has code and examples and the like. That makes it good.

That said, why the title here? Because again, I want to emphasize what your boss really cares about is business continuity. At the end of the day they want to know, “if our server crashes, can we recover?” And the answer had better be “Yes.” This means that you need to be able to restore those backups, Or have another form of recovery.

Log-Shipping

It seems to me that over the years log-shipping has sort of fallen out of favor. “Oh we have SAN snapshots.” “We have Availability Groups!” “We have X.” “No one uses log-shipping any more, it’s old school.”

In fact this recently came up in a DR discussion I had with a client and their IT group. They use a SAN replication software to replicate data from one data center to another. “Oh you don’t need to worry about shipping logs or anything, this is better.”

So I asked questions like was it block-level, file-level, byte-level or what? I asked how much latency there was? I asked how we could be sure that data was hardened on the receiving side. I actually never got really clear answers to any of that other than, “It’s never failed in testing.”

So I asked the follow up question, “How was it tested.” I’m sure their answer was supposed to reassure me. “Well during a test, we’d stop writing to the primary, shut it down and the redirect the clients to the secondary.” And yes, that’s a good test, but it’s far from a complete test. Here’s the thing, many disasters don’t allow the luxury of cleaning stopping writes to the primary. They can occur for many reasons, but in many cases the failure is basically instantaneous. This means that data was inflight. Where in flight? Was it hardened to the log? Was that data in flight to the secondary? Inquiring minds want to know.

Now this is not to say these many methods of disk based replication (as opposed to SQL based which is a different beast) aren’t effective or don’t have their place. It’s simply to say, they’re not perfect and one has to understand their limitations.

So back to log-shipping. I LOVE log-shipping. Let me start with a huge caveat. In an unplanned outage, your secondary will only be up to date as the most recent log backup. This could be an issue. But, the upside is, you should have a very good idea of what’s in the database and your chances of a corrupted block of data, or the like is very low.

But there’s two facts I love about it.

  1. Every time I restore a log file, I’ve tested the backup of that log file. This may seem obvious, but, it does give me a constant check on my backups. If my backups fail for any reason, lack of space, a bad block gets written and not noticed, etc. I’ll know as soon as my next restore fails. Granted, my FULL Backups aren’t being restored all the time, but I’ve got at least some more evidence that my backup scheme in general is working. (and honestly, if I really needed to, I could backup my copy and use that in a DR situation.)
  2. It can make me look like a miracle worker. I have, in the past, in a shop where developers had direct access to prod and had been known to mess up data, used log-shipping to save the day. Either on my DR box, or a separate box I’d keep around that was too slow CPU wise for DR, but had plenty of diskspace, I’d set it to delay applying logs for 3-4 hours. In the event of most DR events, it was fairly simple to catch-up on log-shipping and bring the DR box online. But more often than not, I used it (or my CPU weak but disk heavy box) in a different way. I’d get a report from a developer, “Greg, umm, I well, not sure how to say this, but just updated the automobile table so that everyone has a White Ford Taurus.” I’d simply reply, “give me an hour or so, I’ll see what I can do.” Now the reality is, it never took me an hour. I’d simply look at the log-shipped copy I had, apply any logs I needed to catch up to just before their error, then script out the data and fix the data in production. They were always assuming I was restoring the entire backup or something like that. This wasn’t the case, in part because doing so would have taken far more than an hour, and would have caused a complete production outage.

There was another advantage to my 2nd use of log-backups. I got practice at manually applying logs, WITH NOROLLBACK and the like. I’m a firm believer in Train as you Fight.

Yes, in an ideal world, a developer will never have such unrestricted access to Production ( and honestly it’s gotten better, I rarely see that these days) and you should never need to deal with an actual DR, but we don’t live in an ideal world.

So, at the end of the day, I don’t care if you do log-shipping, Taryn Pratt’s automated restores or what, but do restores; both automated and manually. Automated because it’ll test your backups. Manually because it’ll hone your skills for when your primary is down and your CEO is breathing down your neck as you huddle over the keyboard trying to bring things back.

Reminder

As a consultant, I’m always looking for new clients. My primary focus is helping to outsource your on-prem DBA needs. If need help, let me know!