I’m writing this from upstate NY, outside of Albany, but from what I can tell, this post could be written almost anywhere in the US right now.
I’ll admit I didn’t sleep as well last night as I’d have liked. I kept waking up expecting to hear the beep of UPSs or other indications we’d lost power. About the only sound I heard was a snow plow go by once or twice.
We’re fine here. Most schools are on winter break as it is, but those which weren’t are closed for the day. Scattered power outages, but I suspect most will be back in a few hours. And it’s supposed to warm up into the 40s later today. I’m not too worried.
But, I’m looking at reports from caving and #sqlfamily friends and coworkers from across the country and it’s grim.
From one: I have about one bar left on my battery and no power.
Just saw another: And then the power went out.
Another had a nice screen shot of her pool in Texas freezing over. I should point out, that’s COLD. Not just “oh it’s 31F cold” but much colder because generally a pool has quite a bit of latent heat in it and even though it’s just the surface freezing, it means that the heat in the water deeper in the pool can’t keep the surface warm.
I’m reading reports of rolling blackouts in Texas and portions of the Rocky Mountain states. For example, Texas hit a new record for a single day demand as everyone turned on any electric heating device they could find.
That said, even for everyone who survives this deep freeze (and let’s be honest and clear here, there are some who won’t. Most of us will never know their names, they’re the ones sleeping under a bridge overpass, living in a trailer on the edge of town without access to better resources, the person living in their car who can’t get to a shelter) the problems are just beginning.
There’s going to be a lot of folks reporting water damage due to burst pipes. There’s going to be a lot of car repairs for folks whose cars have been damaged or even totaled due to accidents.
And I highly suspect there’s going to be a marked increase in energy prices over the next few months. From what I understand, the largest refinery in the US shut down yesterday due to the weather. A number of well heads have frozen, and more. Now, this is not an apocalyptical catastrophe. Most will be up and running in the next few days, but such a disruption will cause ripple effects.
Combine that with the ongoing economic impact of COVID on jobs and the economy in general, let me just say Spring can’t come soon enough.
I’m fortunate and I know that. I’m warm. I have heat and electricity. I have plenty of food in the house and neither my wife nor I have lost work in the past year. But I think and worry about those not as fortunate. I’ve contributed more to charities this year than the past and expect to continue to do so. I’ll continue with my other charity work and continue to advocate for policies I think that will help others that need it.
But I still hope for Spring, both literally and figuratively.
One of the replies I thought completely missed the point and I tried to respond in a somewhat humorous but pointed way to the man replying. It took him about 9 hours, but he finally replied and I think based on his reply, completely missed my point. Oh well. I had tried.
But it got me thinking. Had I been blunt enough? Should I have been a bit more confrontational? Could *I* stand up at a conference and give Dr. Gunter’s pointed reply? And of course, the nagging question in the back of my head, “had I commented when I a question was the appropriate response?” or otherwise ended up “centering the discussion around me rather than the original person?”
Let me address the last first. I’m sure I have. I like to think “well I’m a friendly guy, I like to relate and show I’m relatable.” And that’s all true, but, that’s also part of the problem. It’s a case of recentering a discussion or something around me. I’ve always tried to be conscious of this since college when I took a class at the women’s college down the hill and realized that what I had heard about men dominating discussions was true. In a class of about 20 students, with just 3 men, one of the men (and no, it wasn’t me) clearly dominated the discussion.
Anyway, back to my response. I actually sometimes am jealous of some of my friends who can have “I have no fucks to give” attitude and will openly confront someone like that. I think sometimes that can be a good thing, especially with a more egregious example. And I’ve seen some that are pretty bad.
My attempt to deflect the one tweeter’s reply with a bit of humor apparently failed. So I started to think about how I might handle this at an actual seminar and then I realized I had.
It was at a SQL Saturday a number of years ago. It was a good topic, though, for reasons unknown to me, the presenter has not, to my knowledge presented again. And then there was the raised hand. It was someone I knew. And, he had more of a question than a comment. Then again about 10 minutes later. And I think probably a 3rd time. Now, he was in now way being mean or malicious. Heck, I think no matter how hard you looked, you’d never find a mean bone in him. He’s genuinely a decent guy.
But, and this is what I think we all need to do, after the talk, I pulled him aside and pointed out what he had done. He as embarrassed and apologetic. And he vowed to do better.
And as I write that, I realize, this happened TO me. See, I said I wasn’t perfect and I had failed. This time it was on Twitter. I typed a reply that I meant to be supportive and add a touch of humor. A friend DM’d me, “Really?” At first I was confused, but when I asked for her to expand, she pointed out what I had done. Yes, I had meant well, but sometimes intentions are less important than results or even perceptions. I decided to delete my comment, despite her saying it wasn’t necessary. I realized I had not contributed to the discussion and my comment could be a distraction that wasn’t needed. And since them I’ve tried to be better. But a comment she said stood out to me. She DMd me because she thought I was one of the “good ones” that I’d listen and accept feedback. That meant a lot to me. She could have ignored my comment and let me continue to be a jerk at times, or she could have publicly called me out and humiliated me, which might made her point publicly, but caused me to be hurt and not grow. She took the time. I appreciate that.
However, yesterday’s Twitter thread reminded me that all too often in situations like this, women and other minority (in that environment) group end up doing the emotional labor of trying to keep the discussion from recentering the discussion the “I have more of a comment than reply” crow.
Therefore, I think often the onus needs to be on us men to call out our fellow men to say, “hey, that’s not cool” or “do you realize how you came across there? I know you didn’t mean that.” We can’t rely on women and other minority groups to do all the emotional labor. So if you see someone trying to talk over a speaker, pull them aside. If you hear them make an off-color comment in a meeting, speak up. Call out behavior. Find a method that works for you.
I prefer, but am not always good about doing it, calling our behavior a bit more publicly. Not necessarily to embarrass the commenter, but to hopefully get them to correct their behavior and so that the original speaker knows they have support.
For example, if someone in a meeting makes a comment about “yeah, let the girls over in accounting handle it”, unless this is an accounting class for teenagers at an all-girls school, you can and should say “Umm, you mean the women right?”
You don’t have to humiliate a person to make the point. In most cases, the person doing it may not be aware and simply needs a nudge. Give them that chance like I was given. Now, in the end, there will be a few folks that do need to be simply called out and made an example of. I’m ok with that, but for the vast majority let’s work to give them the nudge.
Hunter S. Thompson once said, “Anything worth doing, is worth doing right.”
There’s been some pushback on this that I think has merit with some folks saying “Anything worth doing, is worth doing poorly.” In other words, sometimes attempting something, and perhaps not fully succeeding is still probably worth doing. If I decide I want to bake some bread, but I’m convinced I need to use only artisanal hand-ground flour and yeast I’ve harvested myself to do it “right”, I may never get a sandwich. But if I’m willing to settle for some cheap white flour and yeast I bought at the store, I’ll probably get that loaf baked a lot sooner. It may be a poor substitution for my original goal, but I’m at least no longer hungry.
But, I’d argue, even then it’s worth making that loaf well and with some care.
When I first started this blog, one of my goals was to focus on how I approach problems and look into thought processes. Yesterday I was reminded of this. It was a small ask, a process to monitor if new rows were being inserted into a database daily. If there were no rows, it meant the logging process had broken and probably needs to be recycled. Now, let me be clear from the start, that this is not a life critical process. It is right now, much more a “nice to know” process.
So, first I wrote a query, nothing special, basically:
declare @count int
select @count=count(*) from bar.dbo.report_logs where date = cast(getdate() as date) if @count = 0 exec msdb.dbo.sp_send_dbmail @recipients=’email@example.com’, @subject=’No rows’, @body=’check out the foo process!’
Now, my first instinct was to put that into a scheduled task and be done. But then I thought, “this should really be a stored procedure”. So I wrapped it in some code to turn it into a stored procedure.
Then I realized that I should really probably at least put a few comments in, namely who wrote it, when, and most importantly why. While it’s obvious WHAT it does, it wasn’t clear why we’d want this.
It’s at that point that I had the thought that if I were going to do this, I’d do it well.
So, what’s the difference between doing this well and right? In a more perfect world, I might actually have this process cycle the service in question. But then I’d have to have code to handle situations where the service doesn’t come up. And we’d have to file a change order to introduce a process that potentially would have an impact on a service. And we’d have to wait until the end of the month freeze window, and, well the list goes on.
So, while I’m not going to do all of that, I’m going to make sure the part I can and will do is done well, commented properly, and documented properly. The rest can wait until we really need it.
If you are going to do something, even if you do it poorly, do that part well.
I’ll admit, I was sitting here, struggling for a topic when I received an email alerting me to Deborah Melkin’s latest blog post. So, to take a phrase from Pablo Picasso “Lesser artists borrow; great artists steal”. To be clear, in this case, I’m borrowing an idea from Deborah, a great artist, I’m not. At least not in this case.
In the past few months I’ve come to realize how much I missed travel. I think many of have. But instead of listing real places I want to visit, I’m going to change things a bit and talk about a few fictional places I might enjoy visiting.
Firstly, Middle-Earth. Some may know that my twitter handle is @stridergdm. This came out of my youth of hiking parts of the Appalachian Trail. It’s common when signing into logbooks to adopt a trail name. Being a huge fan of The Lord of the Rings, I adopted Strider. I still use it. But online, I found Stridergdm was pretty much guaranteed to be unique. So, back to Middle-Earth. It is still one of the richest most fleshed out fictional worlds and there’s too much to see. But I suspect I would want to ride the plains of Rohan as dawn rose over the horizon. Sail up Anduin towards Osgiliath in its prime. Perhaps look over the fallen foundation of Barad-dur and then travel back to the First Age and see Gondolin. (yes, I’d want a bit of time travel too.) But simply put there is so much to see there.
Earth-Sea – To sail from the Gont to Havnor and visit some of the islands in between. I always found Ursula Le Guin a unique world, a world of only islands, no continents.
Dune – deserts have always fascinated me, but to visit a world that is entirely desert and life is focused on the lack of water intrigues me. Perhaps to enter Sietch Tabr in the evening as the Fremen are waking up, with the sunsetting at my back, casting my long shadow ahead of me. To break bread and share water with Stilgar and others. Then leave on a Highliner and visit the polar opposite, the ocean world of Caladan.
Mid-World – Of Stephen King’s The Dark Tower series. This is very much a broken world, not just the buildings of Lud, but physics itself where sometimes the Sun doesn’t rise in the East and time seems to flow strangely. But to perhaps visit the Halls of Gilead in their prime and then ride The Drop and overlook the Clean Sea, but finally to walk among the field of roses at Can’-Ka No rey and enter the Dark Tower itself.
Ringworld – it may be unstable, but is probably arguably the largest “world” of any I would want to visit. To look up at the Sun, always at Noon and to realize one could walk multiple lifetimes and still barely make their way around the world. It’s big enough that it has 1:1 maps of Earth and Mars.
Narnia – C. S. Lewis and Tolkien were comrades in arms and members of the inklings but had very different ideas of world-building and different ideas on the proper place for allegory. Tolkien was never a huge fan of Lewis’s Narnia because it hit the read over the head too much with allegory, but I still think it would be interesting to sail on the Dawn Treader to the edge of the world.
One Hundred Acre woods – Technically this is based on a real woods and one could visit it, but not with Pooh and Piglet. This place has a special place in my heart. It’s the first “real” book I ever owned. As a young child, I want to say 3 or 4, a friend of my father’s loaned it to me. More accurately to my father to read to me, but that’s just technicality. After it was read, I tried to return it as I had been taught, after all, it wasn’t mine, it was only loaned. But the owner (who I can’t recall) looked at me gravely and asked me if I enjoyed it and I affirmed I did. “It’s yours then. Keep it.” And I did. And I have since given it to my son.
Stephen King’s Maine – this is a fascinating world filled with a town taken over by vampires and where Mrs. Toad can find shortcuts. What else wonderful and scary things could be found there?
There are almost certainly other worlds, but I think I’ll stop here. While I can never visit any of the above places in reality, the beauty is I can visit time and time again anytime I pick up a book.
This post is the result of several different thoughts running through my head combined with a couple of items I’ve seen on social media in the past few days. The first was a question posted to #SQLHelp on Twitter in regards to if a DBA came into a situation with a SQL Server in an unknown configuration what one would do. The second was a comment a friend made about how “it can’t get any worse” and several of us cheekily corrected him saying it can always get worse. And of course I’m still dealing with my server that died last week.
To the question of what to do with an unknown SQL Server, there were some good answers, but I chimed in saying my absolute first thing would be to make backups. Several folks had made good suggestions in regards to looking at system settings and possibly changing them, possibly re-indexing, etc. My point though was, all that could wait. If the server had been running up until now, while fixing those might be very helpful, the lack of fixing things would not make things worse. On the other hand, if there were no up to date backups and the server failed, the owner would be in a world of hurt. Now, for full disclosure, I was “one-upped” when someone pointed out that assuming they did have backups, what one really wanted to do was a restore. I had to agree. The truth is, no one needs backups, what they really need are restores. But the ultimate point is really the same, without a tested backup, your server can only get much worse if something goes wrong.
I’ve had to apply this thinking to my own dead server. Right now it’s running in a Frankenbeast mode on an old desktop with 2GB of RAM. Suffice to say, this is far from ideal. New hardware is on order, but in the meantime, most things work well enough.
I actually have a newer desktop in the house I could in theory move my server to. It would be a vast improvement over the current Frankenbeast; 8GB of RAM and a far faster CPU. But, I can’t. It doesn’t see the hard drive. Or more accurately, it won’t see an OS on it. After researching, I believe the reason comes down to a technical detail about how the hard drive is setup (namely the boot partition is what’s known as a MBR and it needs to be GPT). I’ll come back to this in a minute.
In the meantime, let’s take a little detour to mid April, 1970. NASA has launched two successful Lunar landings and the third, Apollo 13 is on its way to the Moon. They had survived their launch anomaly that came within a hair’s breadth of aborting their mission before they even made orbit. Hopes were high. Granted, Ken Mattingly was back in Houston, a bit disappointed he had been bumped from the flight due to his exposure to rubella. (The vaccine had just been released in 1969 and as such, he had never been vaccinated, and had not had it as a child. Vaccines work folks. Get vaccinated lest you lose your chance to fly to the Moon!)
A routine mission operation was to stir the oxygen tanks during the flight. Unfortunately, due to a Swiss Cheese effect of issues, this nearly proved disastrous when it caused a spark which caused an “explosion” which blew out the tank and ruptured a panel on the Service Module and did further damage. Very quickly the crew found themselves in a craft quickly losing oxygen but more importantly, losing electrical power. Contrary to what some might think, the loss of oxygen wasn’t an immediate concern in terms of breathing or astronaut health. But, without oxygen to run through the fuel cells, it meant there was no electricity. Without electricity, they would soon lose their radio communication to Earth, the onboard computer used for navigation and control of the spacecraft and their ability to fire the engines. Things were quickly getting worse.
I won’t continue to go into details, but through a lot of quick thinking as well as a lot of prior planning, the astronauts made it home safely. The movie Apollo 13, while a somewhat fictionalized account of the mission (for example James Lovell said the argument among the crew never happened, and Ken Mattingly wasn’t at KSC for the launch), it’s actually fairly accurate.
As you may be aware, part of the solution was to use the engine on the Lunar Module to change the trajectory of the combined spacecraft. This was a huge key in saving the mission.
But this leads to two questions that I’ve seen multiple times. The first is why they didn’t try to use the Service Module (SM) engine, since it was far more powerful and had far more fuel and they in theory could have turned around without having to loop around the Moon. This would have saved some days off the mission and gotten the astronauts home sooner.
NASA quickly rejected this idea for a variety of reasons, one was a fairly direct reason: there didn’t appear to be enough electrical power left in the CSM (Command/Service Module) stack to do so. The other though was somewhat indirect. They had no knowledge of the state of the SM engine. There was a fear that any attempt to use it would result in an explosion, destroying the SM and very likely the CM, or at the very least, damaging the heatshield on the CM and with a bad heatshield that would mean a dead crew. So, NASA decided to loop around the Moon using the LM descent engine, a longer, but far less risky maneuver.
Another question that has come up was why they didn’t eject the now dead and deadweight, SM. This would have meant less mass, and arguably been easier for the LM to handle. Again, the answer is because of the heatshield. NASA had no data on how the heatshield on the CM would hold up after being exposed to the cold of space for days and feared it could develop cracks. It had been designed to be protected by the SM on the flight to and from the Moon. So, it stayed.
The overriding argument here was “don’t risk making things worse.” Personally, my guess is given the way things were, firing the main engine on the SM probably would have worked. And exposing the heatshield to space probably would have been fine (since it was so overspecced to begin with). BUT, why take the risk when they had known safer options? Convenience is generally a poor argument against potentially catastrophic outcomes.
So, in theory, these days it’s trivial to upgrade a MBR disk to a GPT one. But, if something goes wrong, or that’s not really the root cause of my issues, I end up going from a crippled, but working server to a dead server I have to rebuild from scratch. Fortunately, I have options (including now a new disk so I can essentially mirror the one disk, have an exact copy and try the MBR->GPT solution on that one) but they may take another day or two to implement.
And in the same vein, if it’s a known SQL Server, or an unknown one, you’re working on, PLEASE make backups before you make changes, especially anything dramatic that risks data loss. (and I’ll add a side note, if you can, avoid restarting SQL Server when diagnosing issues, you lose a LOT of valuable information in the DMV tables.
So things CAN get worse. But that doesn’t mean there’s any need to take steps that will. Be cautious. Have a backout plan.
I’ll admit I was tempted to just have a blank page here and say I’m starting 2021 with a blank slate. But that seemed too easy.
But with hope in hand, I will set some goals:
Continue to write for Red-Gate! I enjoy this and have to admit, the little extra spending money isn’t so bad.
Expand my client base. A concern I’ve had for a few years, and 2020 really reminded me of this, too much of my current consulting depends on one large customer. I want to change that. I’ve already reached out to some colleagues in regards to possible clients and will continue to do so. Part of my goal is to really define my business model here.
Have a successful NCRC Weeklong Cave Rescue Seminar here in NY. This was originally planned for 2020, but in light of the risk of becoming a super-spreader event, we postponed. Right now it’s still not clear if we’ll be able to safely host the event, but the odds are slowly improving. I’ve been monitoring my Facebook feed and it’s amazing how many folks I know already who are in the process of getting vaccinated against Covid-19. If this continues at this rate, we’ll be able to have an event!
Continue blogging. 2020 was my best year year, but I hope to improve in 2021. Of course I have to admit my numbers were helped when Brent Ozar mentioned one of my posts. So, I guess the real secret is to get Brent to retweet my blogs! Seriously though, I enjoy blogging and as much as I often do it for others, I’ll admit, some of it is really just vanity for me, but also serves as a practice to keep up on my writing and creative skills.
Travel! Once it’s safe to travel again, I hope to do a LOT more this year. One benefit of being an independent IT consultant is I basically can work any place I have a steady Internet connection. I hope to take advantage of that.
Continue biking. This probably means finally getting a new bicycle. I had hoped to replace my Trek 520 that is now 30 years old this year, but due to Covid basically avoided bicycle shopping. But after 1300 miles last year, I think it’s time to seriously consider a new one.
Hike more. I loved my overnight trip last year. I hope to complete the section of the Appalachian trail in Massachusetts I haven’t done yet and then perhaps finish Connecticut or Vermont.
Continue to enjoy life. I mentioned yesterday that as crappy as a year it was, I actually enjoyed much of the year. If anything, Covid forced me to take stock and focus on enjoying life. I want to continue to do this.
And, most of all, make it through another year with you all.
Yes, I’m joining the chorus of so many others who are publishing a lookback on the previous year. This has become a tradition for me. And I of course followed last year’s review with a preview for 2020. I made the obligatory dad joke then and I’ll make one now, that I can’t wait until 20/20 is truly hindsight!
2020 I think upset everyone’s goals, and mine were no different. But I figured I’d start with my goals from last year and then try to end on an actual up-note.
I had a goal of blogging at least once a week. I think I missed 1 this year, but a few weeks I blogged more than once, so, including this post, I will have 56 posts this year. Not to shabby. And my overall page count is up. So that’s good.
I vowed to write more for Red-Gate and I did. But not as much as I’d like. I do blame this partly on Covid. I lost some of my enthusiasm. But I am working on another article. I was hoping to have it done this week, but lost motivation. I did learn one of my articles there is one of their top read articles. I’m quite proud of that!
I did read more this past year, that’s for sure.
One goal I had was to keep speaking at more SQL Saturdays. Well, that didn’t quite go as planned. I did speak at the Albany event, but that was about it. This one I 100% blame on Covid. On the other hand, I finally attended the Portland Oregon SQL Saturday, albeit virtually.
Speak at SQL Summit: well I did achieve this one, sort of. It was virtual, but I was selected and that was a HUGE highlight. And in fact I ended up being part of two presentations, the 2nd a live one that I ended up doing from my car while waiting for something else. And my presentation on PowerShell for Beginners apparently was very popular. So, I can at least say I went out on a high note.
Started to use git on a far more regular basis, including from the command line (previously I had limited myself to the GUI in Visual Studio).
I did read more! – including:
The Power Broker, I biography of Robert Moses
Station Eleven, though in retrospect, reading a book about the world after a global pandemic was NOT a great way to start the year!
So, overall, I did accomplish a number of my goal. I had some generic ones that included caving more, biking more, and hiking more. More on those in a moment.
Overall, the year was a bummer in many ways. I really missed travelling. I really missed seeing friends and family (I think we saw my mom in person twice during the entire year). I missed my seeing my #SQLFamily in person. I missed my NCRC Family. I missed having our normal annual pool party.
I missed, normalcy.
You know what, for me personally, 2020 was actually a year of some ups. I’ve been very fortunate and I was able to do things I had not done as much as in the past.
For one I accomplished my first overnight hike in perhaps over a decade, a nice 18 miles on the Appalachian Trail. Due to scheduling I couldn’t quite get in a 2nd hike that would have completed a gap in Massachusetts, but perhaps next year. I did find I needed some new equipment, which I purchased in anticipation for next year.
As for biking, I definitely did a LOT more this year. I biked over 1300 miles this year (I can’t recall the last time, if ever, I did as much) including my first century ride (100 miles in one day) since my senior year of high school. I’m really proud of that one. Who knows, maybe I’ll do another in 2021.
Despite a contentious election season, America voted the Orange One out.
Spent more time with my family! We did several walks around the area including some paths we had not explored in the 20+ years my wife and I have lived in the house.
I fixed the dryer that had been making a horrible rumbling sound for years.
I made a bunch of sourdough bread, pancakes, waffles and even a pizza crust or two. And I’m back to making sourdough again.
I finished binge-watching Haven, a quirky fun show based on a Stephen King story.
I rewatched (and for my family, they watched for the first time) the entire Prisoner series. Yes, everyone still wonders what the hell the final episode is still about.
We’re binge-watching Schitt$ Creek and wondering why we didn’t watch it sooner.
All in all, it was a very mixed year. It wasn’t a normal year in many ways and some of the normalcy, I really missed. But, on the flip side, I think it encouraged and in some cases forced me out of my comfort zone and to do things I might not have had time to do otherwise.
I can’t say it was a great year. While I’ve been fortunate and have not had anyone close to me succumb to Covid, I know too many people who have had friends and family die of it. So in that aspect, it’s been a terrible year. As of the latest count, over 342,000 have died in the US and yesterday set a new record in the US for daily deaths. At this rate, by the start of next week we’ll have over 350,000 deaths and predictions are of over another 80,000 in the next three weeks. PASS has been a side casualty of this too.
Too many lives have been impacted and effected and I don’t want to minimalize those.
But, I do want to highlight that even in a dark year, at least personally, I’ve been able to find some positive things to focus on. I hope others have too. Hopefully everyone reading this has at least one thing they can look back on and say, “Yeah, that was good” or “That’s a special memory, I won’t forget.”
And to quote Colonel Potter from MASH: “Here’s to the New Year. May she be a damn sight better than the old one, and may we all be home before she’s over.”
I wrote once before about a day being a “Monday” and a week later about it not being a “Monday”. Well, yesterday was another Monday. And it reminded me of the value of DR planning and how scaling to your actual needs and budget are important.
There’s an old saying a Cobbler’s Children has no shoes and there’s some truth to that. And, well my kids have shoes, but yesterday reminded me I still want to improve my home DR strategy.
I had actually planned on sleeping in late since it’s the week between Christmas and New Years and my largest client is basically doing nothing and everyone else in the house is sleeping in this week. But that said, old habits die hard and after one of the cats woke me up to get fed, I decided to check my email. That’s when I noticed some of the tabs open in Chrome were dead. I’m not sure what I looked at next, but it caused me to ping my home server: nothing.
While that’s very unusual, it wouldn’t be the first time it did a BSOD. I figured I’d go to the basement, reboot, grab the paper, some breakfast and be all set. Well, I was partly right. Sure enough when I looked at the screen there was an error on it, but not a BSOD, but a black and white text screen with a bunch of characters and a line with an error on it. I rebooted, waited for the Server 2012 logo and then went out to get the newspaper. I came back, it was still booting, but I decided to wait for it to complete. Instead, it threw another BSOD (a real BSOD this time). I did a reboot and seconds later up came a BIOS message “PARITY ERROR”.
I figured it must be a bad RAM chip and while 16 GB wouldn’t be great, I could live with that if I had to cut down. But, things only got worse. Now the server wouldn’t even boot. I don’t mean as in I kept getting parity errors or a BSOD but as in, nothing would happen, no BIOS, nothing. Best as I can tell my server had succumbed to a known issue with the motherboard.
The technical term for this is “I was hosed”. But, in true DR spirit, I had backup plans and other ideas. The biggest issue is, I had always assumed my issue would be drive failure, hence backups, RAID, etc. I did not expect a full motherboard failure.
On one hand, this is almost the best time of the year for such an event. Work is slow, I could work around this, it wouldn’t normally be a big issue. However, there were some confounding issues. For one, my daughter is in the midst of applying to colleges and needs to submit portfolio items. These are of course saved on the server. Normally I’d move the server data drive to another machine and say “just go here” but she’s already stressed enough, I didn’t want to add another concern. And then much to my surprise, when I called ASRock customer service, they’re apparently closed until January! Yes, they apparently have no one available for a week. So much for arguing for an RMA. And finally of course, even if I could do an RMA, with the current situation with shipping packages, who knew when I would get it.
So, backup Plan A was to dig out an old desktop I had in house and move the drives over. This actually worked out pretty well except for one issue. The old desktop only has 2 GB of RAM in it! My server will boot, but my VMs aren’t available. Fortunately for this week that’s not an issue.
And Plan B was to find a cheap desktop at Best Buy, have my wife pick it up and when she got home, move the server disks to that and have a reasonably powered machine as a temporary server. That plan was great, but, for various reasons I haven’t overcome yet, the new machine won’t boot from the server drive (it acts like it doesn’t even see it.) So, for now I’m stuck with Plan A for now.
I’ve since moved on to Plan C and ordered a new Mobo (ironically another ASRock, because despite this issue, it’s been rock solid for 4+ years) and expect to get it by the 5th. If all goes well I’ll be up and running with a real server by then, just in time for the New Year.
Now, Plan D is still get ASRock to warranty the old one (some people have successfully argued for this because it appears to be a known defect). If that works, then I’ll order another case, more RAM and another OS license and end up with a backup server.
Should I have had a backup server all along? Probably. If nothing else, having a backup domain controller really is a best practice. But the reality is, this type of failure is VERY rare, and the intersection of circumstances that really requires me to have one is more rare. So I don’t feel too bad about not having a fully functional backup server until now. At the most, I lost a few hours of sleep yesterday. I didn’t lose any client time, business or real money. So, the tradeoff was arguably worth it.
The truth is, a DR plan needs to scale with your needs and budget. If downtime simply costs you a few hours of your time coming up with a workaround (like mine did), then perhaps sticking with the work around if you can’t afford more is acceptable. Later you can upgrade as you needs require it and your budget allows for it. For example, I don’t run a production 24×7 SQL Server, so I’m not worried about clustering, even after I obtain my backup server.
If you can work in a degraded fashion for some time and can’t afford a top-notch DR solution, that might be enough. But consider that closely before going down that route.
On the other hand, if like my largest client downtime can cost you thousands or even millions of dollars, than you had darn well invest in a robust DR solution. I recently worked with them on testing the DR plan for one of their critical systems. As I mentioned, it probably cost them tens of thousands of dollars just for the test itself. But, they now have a VERY high confidence that if something happened, their downtime is under 4 hours and they would lose very little data. And for the volume of business, it’s worth it. For mine, a few hours of downtime and a few days of degraded availability is ok and cost effective. But, given I have a bit of extra money, I figure it’s now worth mitigating even that.
In closing because this IS the Internet… a couple of cat pictures.
First, I want to thank Rie Irish for giving me cause to blog a second time this week. Normally this time would be sent writing up another article for Redgate (yes Kathi I’ve got one more in the works).
Second, I want to add, that up until now I had decided to refrain from writing about the fall of PASS. I knew eventually I probably would, but some good news last night has accelerated that process.
Last night Rie posted to Twitter an announcement about a new initiative by Microsoft to provide a space and tools for the former PASS User Groups. I was thrilled when I saw this and I’ll explain why.
In the past few days there has been a LOT of efforts put forth by various members of the #SQLFamily to create new spaces for the former members of PASS, both individually, and as groups. I’ll admit, I was both heartened by this, but also a little concerned. I was obviously heartened, because as I knew would happen, the members of #SQLFamily have stepped up and tried to fill various needs. This is a great volunteer community! I want to emphasis that. Even weeks ago when I had suggestions forwarded to me that PASS was about to close shop, I felt that the community would go on.
However, my concern was that it would fracture, that various little domains and fiefdoms would develop. Now, this may not necessarily be the worst thing that could happen. A decentralized community might actually be a good idea. For example if we separate the concept of Saturday events from User groups, that might spur innovation and might encourage new ideas to come forth. But, it might also inhibit things for folks who have organized both if they end up having to maintain two mailing lists, two sites, etc. So it’s a mixed bag. That said, if nothing else had happened, I’d take that over having the community completely falling apart.
And even if nothing else had happened, I know my User Group has speakers scheduled out through May of 2021 (and the only reason I haven’t scheduled beyond that is because I’m hoping by June to know if we can do in-person again and if so when). I know that Monica Rathbun has the Hampton Roads SQL Server User Group scheduled out through all of 2021. This is the case with a number of other User Groups. So, no matter what, User Groups would continue.
And there are still SQL Saturdays in the works, by that name or others. So, the community will continue.
But, as I said, my concern was the community might fracture into fiefdoms.
I think the announcement from Microsoft goes a LONG ways to allaying my fears. Yes, it’s just a press announcement with details to be worked out, but it has several things going for it. The biggest is the name behind it: Microsoft. Let’s be honest, the one thing the #SQLFamily has in common is we all work on the Microsoft Data Platform. So, to me, it makes sense that Microsoft be a central focus. Another is it appears this platform will end up providing a lot of the tools community leaders will need, and all in one place. These two facts should help keep the community from fracturing.
Now, I already know there is going to be one huge objection from some: “But it’s Microsoft, they can dictate what we do! We want something independent like PASS was!”
Well, first let me point out, for all its independence, PASS is no more. So independence is no guarantee of perpetual success.
Secondly, please read the release with an open mind. It’s NOT an attempt to recreate PASS. There’s no talk of a Board of Directors, there’s no talk of a main event. It’s very clear, at this stage what it is: a set of centralized resources to sustain the community. Additionally I’ve spoken further with Rie that has given me further confidence in the plan. We will see more over time how the winds blow, but I am comfortable recommending moving forward with this path. And you know what, if it turns out the Microsoft suddenly wants to take things in another direction, #SQLFamily will again do what it needs to for its members.
So, this is NOT “PASS Version II”. It doesn’t attempt to be. Perhaps PASS Version II will come to pass someday. I sort of hope it does. I look forward to another Summit. But for now, I think this is an excellent step forward. I will end by pulling a quote from the release:
Although Microsoft built SQL Server, it’s clear that the passion and dedication from each of you is what makes it thrive.
And I think this is as true today as it was 2 weeks ago.
Twas the night before deploy, when all through the server, not a script was running, not with any ferver
The queries were pushed to github with care, in hopes that St. Deployment would soon be there;
The devs were all sitting at home on couches, while visions of good deploys proved they were no slouches;
And Mindy in her ‘cubicle and I in my car, Had just submitted our Java in one big jar;
When out on the floor there arose such a clatter, I sprang from my seat to see what was the matter;
Away to windows I flew with a dash, Tore open the login only to see the OS crash.
The icon on the desktop of was blinking just so, gave a lustre of failure to the objects below;
When what to my wonderings eyes should appear, But a minutia of code and eight tiny beer, With a developer so lively and quick, I knew in a moment it must be my buddy Nick.
More rapid than eagles his fingers they flew, and I knew more #SQLFamily was due, Now Rie, Now Kathie, now Andy, now Hamish, On Fritchey, on, Argenis, on Deborah, and on Klee!
To the top of the script, to the top of the code! Now code away, code away, code away mode!
As leaves before the hurricane they fly, When they met a debug, threw it to the sky;
So up to the keyboards the coders they flew With their brains full of tips, and my buddy Nick too;
And then in a twinkle, I heard on the floor, the opening and closing of every door.
As I drew in my head and was turning around, Down the aisle my buddy Nick came with a bound.
He was dressed all in schwag, from his head to his feet, And all his clothes were from Summit and Pass to beat;
A bundle of con gear he had to tell you the truth, And he looked like a vendor just opening his booth.
His eyes — how they twinkled, his demos how merry, His patter, purple prosy, his pitch too cheery!
His droll little cube was a certain new low, And it was clear he would not win best in show;
The stump of a pen he held tight in his teeth, and the ink circled the printout like a wreath;
He was chubby and plump, right jolly old dev, I laughed when I saw him, as the code did rev;
A wink of his eye and a twist of his head, Soon gave me to know I had nothing to dread;
He spoke not a word, but went straight to his work, finding all the bugs in my code, making me feel quiet the jerk;
And laying his finger aside of his nose, and giving a node, up the aisle he rose;
He sprang from his cube, to his team gave a whistle, And away they all flew, quite like a missile;
But I heard him exclaim here he flew from action Remember parametrize your queries, and to all a good closed Transaction!
– With apologies to Clement Moore (to whom I have no known translation) who apparently wrote the original (and much better) version just a few miles from where I currently sit.
From Wikipedia: The poem was first published anonymously in the Troy, New YorkSentinel on 23 December 1823, having been sent there by a friend of Moore, and was reprinted frequently thereafter with no name attached. It was first attributed in print to Moore in 1837.