Rolling Back

I’ve had a number of thoughts rolling around my head lately and they’re somewhat related so here goes.

I’m going to start with a depressing headline and URL: Civilization will end in 2050. Ok, perhaps that’s more of an alarmist headline than anything else. I posted this elsewhere, and some folks correctly pointed out that this could have been written by Thomas Malthus over two centuries ago.  And it’s true; prognosticators have predicted the fall of humans since probably before recorded history.  How many of us recall in the last century alone predictions of world-wide famine, a world-wide ice age, comets smashing into us and other horrible events.

Some were prevented, such with the Green Revolution. Some were corrected with better data or a better understanding of the data. Some where just… nuts.  That said, some predictions can be made with 100% certainty, even if the bounds on the actual circumstances are fuzzy. I’ve been giving this particular nugget a lot of thought.  Since I’ve passed that magical (at least to humans) mark of a half-century, I’ve given more thought to what I call my expiration date. It’s a fact. At some point I will cease all biological functions and will be dead.  I can’t escape that. Every day the odds increase a very tiny percentage.  So, I am predicting my own death, most likely sometime in the next 1/2 century. There’s a very slight possibility it could be tomorrow, and a much greater chance it could be 40 years from now, and based on current medicine an absolute certainty 80 years from now.  You’ll note I hedged that last one. It’s quite possible in the next few decades we figure out how to extend the human lifespan by decades if not centuries.  But I’m not counting on it. And even then, there are no guarantees I’m not hit by a driver while biking or some other catastrophic event.

That said, right now I’d say the odds are decent I’ll be alive in 2050. And almost certainly my kids will be (and very likely I’ll have grandkids by then).  So, am I worried that the world will end in 2050? Yes and no. For one thing, there’s time between now and then for a lot to happen. But, as we get closer, it’s going to be harder and harder to forestall the impacts of higher levels of CO2 in the atmosphere and oceans. Yes, to be clear, I believe the evidence supports that humans are greatly impacting the climate via CO2 and other emissions.

But I’m also optimistic that we’ll work harder, especially as things get worse, to stave off the most pessimistic scenarios. But it won’t be easy and the longer we wait, the harder it will be.

While anecdotal, I’ve already seen the impacts in my life, earlier springs, caves that once held ice year round no longer do, less snow in the winters, etc. It worries me. But I also still have some hope.

That said, an analogy that’s been rolling around in my head that isn’t 100% perfect but fits me as a DBA.

Many of us in the SQL server community have started a long running transaction, only to realize we’ve brought the server to a halt. In other words, no other user can access their data until our transaction is complete. We can abort the transaction, but if we wait too long, that can be a far worse solution than trying to stop it early on. For the non geeks, in other words, if our transaction is going to update 1 billion records and we realize 5 minutes in that it will take 10 hours, we can abort it and it should take about 5 minutes to rollback to the original state. This means we’ve only brought the server to a halt for about 10 minutes. However, if we ignore the problem, and keep pretending it’ll go away and we wait say 9 hours and then finally decide, “oh what the heck, let’s try to fix the problem now” it might take another 9 hours to rollback. In other words, by waiting to long to resolve the problem, instead of being 10 minutes, it can be 1080 minutes.

In other words, the longer you wait, the harder it can be to recover. I think that’s where we’ll be at by 2050. Whether or not I’m still alive (though I plan to be!)

Marshmallows Part II

I’ll have to admit, I can rarely tell in advance when one of my posts will hit all the buttons and generate views and when it’ll fall flat. But as I don’t always write for my audience, sometimes I write for my own reasons, I can live with that.

So, how to follow-up on a post that didn’t receive many views, write a follow-up post. You can call me a slow learner.

Actually, it’s about learning. Last time I wrote about my microwave and doing a quick experiment with marshmallows to prove it was really dead.  After 2 days without a microwave it was time to get a new one. Of course I couldn’t get what I wanted because the space it had to fit into was limited in size.  That could have been resolved, but would have meant redoing the cabinet space it had to fit into. And if I were going to redo the cabinet space there, I might as well redo the rest of the cabinets. And if I’m going to redo the cabinets, I really need to redo the counters. And very quickly a replacement $100 microwave I can get in an hour would become a 3-week $10,000 kitchen remodel. I opted for the $100 microwave over the one I really wanted.

And the results are shown at the top of post (and below in case the top doesn’t appear)

melted marshmallow picture

10 seconds of marshmallows in the microwave

It’s quite interesting to me. The best heating was beyond the area of the rotating plate.  But this also shows the value of the rotating plate since if there’s a few sports, if I put a something to heat and everything was stationary, it would take forever to heat since there’s little to no microwave energy there. (This can get complex because of the size of the wave and the height of the material, etc.)

Now, I’d have done more experiments, but it seems a certain someone in the house enjoys marshmallows more than I do and had eaten a bunch and this was all I had.

But, I have a working microwave and I’ve proven how important the rotating plate can be (not that I had much doubt).

And that’s science to me; doing experiments and learning.

Oh and about the SQL query I was updating. It’s going into production this week hopefully. I was about to eek out about a 10-20% improvement. Beyond that, not much I could do because it really ends up scanning an entire table, on purpose.  Only so much you can do there.

One last thing: there may not be a post next week because I’ll be teaching at the NCRC weeklong cave training class in Indiana and will have limited internet and time.

Marshmallows

Though I attended RPI, which is generally considered an engineering school, my degree is a BS in Computer Science. I say that because I consider myself more of a scientist than an engineer at times. And honestly, we all start out as scientists, but many of us lose that along the way.

Anyone who has had a small child has observed a scientist in action. No, they’re not in a lab full of test tubes and beakers and flasks giving off noxious smells. But they are in the biggest lab there is, the world. They also don’t necessarily realize it. Nor do parents. But every time they drop a Cheerio, they’re testing gravity.  Fortunately (or unfortunately depending on your point of view) so far every time they’ve managed to prove that gravity works. This is the most obvious example, but when you stop to think about it, much of the first few years of life is all about experimenting. Most of the time it goes well, but sometimes, as a burnt hand will attest, the experiment has a less than ideal outcome.

And it’s the fear of burned hands that leads to parents to utter that common  refrain, “Don’t touch that!” or the variation “Don’t do that!”.  Soon, over time, our experimentation starts to get reined in until we do very little of it. This can be inhibiting.

Years ago I used to teach an “Introduction to Windows” adult education class. It was I believe a 6 week class and I taught several over the course of a couple of years. It didn’t take me long to realize the biggest constraint on the students ability to succeed in the class was that they had internalized “Don’t do that, you might break something.” Once I realized that, half my teaching pedagogy simply became, “Touch that, you won’t break it, and if you do, it’s not a big deal, and if it is, we’ll fix it anyway.” Seriously, more than anything else, I had to encourage most of my students to experiment with the computer.

More recently I realized I had stopped doing as many experiments in my life as I should be doing. About 1.5 weeks ago I attended a Wilderness Medicine Conference a friend of mine had told me about. At the end of the very wet, cold, rainy day, a bunch of us went outside and tried to start a fire. Starting a fire, let alone in such conditions was something most of the students had never done. I had, but not in years. With some effort, and experimentation, including using the outside box of a single serving size package of Fruit Loops, we finally managed to get the fire going.

But this got me thinking. When I go hiking, I carry a tiny ziplock back in my jacket with some firestarting materials. They’re there in case of an emergency. But, the thing is, I had never actually tried them and realized if I didn’t know how well they worked in practice, I couldn’t rely on them in emergency. So, I went outside, and started a fire. And I learned that yes, my materials ARE adequate, but the dryer lint needed to be pulled apart more than I realized. I tried again later in the week, and added the use of a toilet paper roll to form sort of a chimney so the starting fire would draft better. This, and the better pulling of the lint worked even better and a single match was sufficient this time.  This gave me more confidence that in an emergency, in less than ideal conditions I could get an actual fire going.

But, I wasn’t done! Our microwave broke this weekend. But, before I wrote it off, I wanted to make sure it wasn’t a fluke or something else. So, in this case I decided to get a bag of marshmallows and lay them out inside the microwave to see if I was getting ANY energy out of the magnatron. Turns out, nope, nada, nothing. So, today or tomorrow I will be buying a new microwave. But, it was a fun, and later tasty experiment.

Without delving deep into the scientific method here, I’ll say at a simple level, science is about having a hypothesis and testing it. The testing it is important.

To bring this back to SQL. First, you have a hypothesis that your backups will work. Have you tested that hypothesis? If not, do so immediately. Even if they do, you might learn something now that will be important when you have to do it for real. Perhaps you learn the volume your backups are on only has write access. Or perhaps you learn you need to retrieve your encryption keys and the person who controls access to them is on vacation. Or perhaps your RPO is 4 hours and the restore takes 6 hours.  So, experiment.

query plan

Capture of a random query plan

Recently for one client I’ve spent some time experimenting with various changes to help improve the performance of some queries. Not everything I tried worked, but some things did. So, again experiment.

I’m curious what recent experiments you may have done, SQL or otherwise. What were their outcomes?

 

Punditry

We’re all experts on everything. Don’t think so? Go to any middle school or high school soccer game and you’ll be amazed at how many parents are suddenly experts on soccer. It’s also amazing at how many parents are parents of future NCAA Division I scholarship soccer players.

Seriously though, we’re all guilty of this from time to time. I’ve done it and if you’re honest, you’ll admit you’ve done it.

Yesterday the world suffered a loss, the near destruction of Notre Dame.  Early during the fire our President tweeted:

“Perhaps flying water tankers could be used to put it out. Must act quickly!”

As many have pointed out, this was actually a terrible idea. The idea of dropping 100s of kilograms of water onto an already collapsing roof is most likely to do more damage than not. But, while I think it’s easy to mock the President for his tweet, I won’t. In some ways it reminds me of the various suggestions that were made last summer during the Thai Cave Rescue. We all want to help and often will blurt out the first idea that comes to mind.  I think it’s human nature to want to help.

But, here’s the thing: there really are experts in the field (or to use a term I see in my industry that I dislike at times: SME (it just sounds bad) Subject Matter Expert.)

And sometimes, being a SME does allow you to have some knowledge into other domains and you can give some useful insight. But, one thing I’ve found is that no matter how much I know on any subject, there’s probably someone who knows more. I’ve written about plane crashes and believe I have a more than passing familiarity in the area. Perhaps a lot more than the average person. But, there’s still a lot I don’t know and if I were asked to comment by a news organization on a recent plane crash, I’d probably demur to people with far more experience than I have.

Having done construction (from concrete work in basements to putting the cap of a roof on), I again, have more than a passing familiarity with construction techniques and how fire can have an impact. That said, I’ll leave the real building and fire fighting techniques to the experts.

And I will add another note: even experts can disagree at times. Whether it’s attending a SQL Saturday or the PASS Conference itself, or sitting in a room with my fellow cave rescue instructors, it can be quite enlightening to see the different takes people will have on a particular question. Often no one is wrong, but they bring different knowledge to the table or different experiences.

And finally, you know what, sometimes the non-expert CAN see the problem, or a solution in a way that an expert can’t. But that said, at the end of the day, I’ll tend to trust the experts.

And that’s the truth because I’m an expert on punditry.

Redemption

About a year ago I wrote this post: And so it Happened… about my first (and so far only) time I ended up with an empty room at a SQL Saturday. I’ve run into a few other speakers who have had the same experience, so that soothed the bruised ego a bit, but it still left a bit of a mark.

As a result, I set a goal of redeeming myself this year again at the Colorado Springs SQL Saturday. I figured it wouldn’t be that hard to exceed my turnout from last year.  So, I submitted several topics for them to select from and waited. Finally the day came, and I found that I had been selected to speak. There was only one problem. The topic in question was one that while I had submitted, and had a good outline for, I had not actually fully developed into a presentation and was a bit nervous about:
The very Model of a Modern Day Database. I thought it would be a good topic, I just had to develop it.  And of course like any good procrastinator I kept putting off the work. I mean I was making progress, but, well it was slow.

Fortunately, by Friday the 5th, I had run through a complete form of it and had worked out pretty much all the tweaks I wanted and had practiced it a few times to an empty room, you know, just in case of a repeat of last year. Seriously though, I do several run-throughs to make sure I get the timing right and I pretty much know what I was going to say. I’ll let readers in on a little secret, some of the parts of my presentations that look like they’re improvised or impromptu comments or replies, are often rehearsed.

So I felt pretty good going into Saturday.  Then, looking at the schedule, it struck me that my topic was on the System Databases, one of which is known as the TempDB (to my non-SQL readers, that’s a fairly critical database SQL Server uses as sort of a scratch pad) and that a session before lunch (mine was scheduled after lunch) was by Kalen Delany and was an entire hour on just the TempDB. I first heard Kalen speak at SQL Connections conference back in 2005 or so in Orlando and had read a few of her books. To say that she’s well known in the SQL Community and highly respected might be an understatement. Now the impostor syndrome was really starting to kick in! What could little ol’ me say about the TempDB in 15 minutes that would interest people after listening to her?

But then I realized, our topics had a slightly different focus, and while some of our advice was similar (put your TempDB on FAST drives), I covered things in a different way and there would still be something of interest to my attendees. And, it is not a competition after all. Honestly, my goal whenever I teach any topic is to reach at least one student or attendee. If I can get one person to walk away and say, “I learned something” or “That was worth it” I feel like I’ve won. This happened during a week-long cave rescue training course once. On the first day in the field I showed a student a fairly simple but not entirely obvious way to rig a rope. After explaining it to her she looked at me and said, “that’s worth the price of the course right there!”.  I glowed and joked I could now take the rest of the week off; I had achieved my goal.

Anyway, after lunch I was prepared. Lunch was scheduled for 12:30-1:45 and I was in the classroom by 1:40, all setup waiting for folks to show up. And sure enough two people showed up. I was happy. Perhaps not ecstatic, but at least happy I had an audience.  And then two more people showed up, put down their stuff and asked, “mind if we leave this here, we’ll be back.”  I said it was fine, but was a bit confused since the clock was saying 1:44 and I was wondering where they’d be going just before my session started.

But hey, four people, that was four more than last year, even if two weren’t in the room and one of the others admitted they weren’t really a DBA and wasn’t sure if the class was applicable to what they wanted to learn.

At that point, one of the original pair started to shuffle her papers and looked up and said, “you know, it’s weird, the schedule has a 15 minute break between lunch and the first afternoon session. This is supposed to start at 2:00 PM”  I looked and she was right.  As far as I can tell, when the organizers laid out the sessions, they put a 15 minute break between them, and simply did the same for after lunch. This explained why the second pair of people had left with the intent to come back. They wanted good seats for the 2:00 PM start.

Sure enough, by 2:00 PM the room was fairly full and I was off and running. I was in a smaller room than Kalen’s presentation, where she had 40 or more, I had perhaps a dozen. But I was happy and content. And, once it was over, both the room monitor and myself reminded folks to give feedback and this audience was great at that.

A word on feedback. The forms at SQL Saturdays tend to be fairly standard and I think I speak for most presenters when I say, that while it can be gratifying to get all 5s on the top of the form, it’s also kind of useless. But, when folks actually take time at the bottom of the form to give actual written feedback, that’s quite gratifying. This audience gave great written feedback.

I also appreciate feedback in person. At least one person came up afterwards to say, “That was really great, I bet you could do an hour on each System Database.”  So perhaps, I will do an hour presentation on the TempDB someday!

So, I feel redeemed. Due to a variety of reasons it’s unlikely I’ll bid to speak at Colorado Springs next  year, but I’d highly recommend it for anyone in the area. And, if you’re afraid that some other presenter might overshadow you because they’re better known or their topic is similar to yours, don’t despair. Seriously, there’s enough knowledge to go around and enough interest.

 

Shouldn’t that be plugged in?

That was the question a friend of mine in 6th grade asked. As a result I developed what I call the Charlie M. rule after my friend. It was sort of Show and Tell day in 6th grade and we were supposed to talk about our hobbies. I brought in a circle of HO scale track (18″ radius for those interested) and my locomotive (a model GP-38) and some cars and of course the transformer to power it all.

I set it all up in front of the class and dutifully tried to demonstrate it. Nothing moved. I checked to make sure the engine was properly on the tracks: check. I made sure the wires were connected to the transformer: check. I made sure the wires were connected to the track: check.  I was stumped: check. Finally Charlie raised his hand and asked, “Shouldn’t that be plugged in?”  Ayup, in all my nervousness and being hurried, I had forgot the most basic step, of plugging in the transformer.

I try to keep this in mind when troubleshooting: check the obvious. I ran into this again over the weekend when trying to get my BMW Z3 running again. (Side note: no, consulting does not pay that well. This is one of the few tangible items I have left from my dad’s estate). It had stopped running late last fall and at the time I spent a little time trying to make it run, without much success. Finally, with the family’s help I pushed and pulled it into a shelter for the winter and then left it for the winter.

I wasn’t planning on worrying about it until later this month, but then… well let’s just say when I put the large box with metal corners into the rear of the Subaru, I forgot to check the obvious and slammed the rear hatch down on the box. Well, the box, realizing it didn’t have enough room, decided to take advantage of the metal corner and proceeded to make more room by punching out the rear window of the Subaru. Oops.  Such a simple mistake, but a large one.

So, while waiting for the Subaru to get fixed, I decided it was time to get the BMW on the road.

Now due to the symptoms, I knew it wasn’t a dead battery or bad gas. So taking advantage of what I call my extended brain, I asked others for help.  We had narrowed the problem down to either the clutch interlock switch or the starter. Neither looked like it would be an easy self-service and I was getting frustrated. I finally decided that perhaps checking the ODB-II codes might yield more information. Strangely though, the reader didn’t power up; there were no codes to read. That struck me as a strange. So here I did check the obvious: I took the reader to the Subaru and made sure the reader worked. And it worked fine on the Subaru. I went back to my extended brain and mentioned that.

“Oh, have you checked the fuses?”

“Nah I thought about it, but everything seems to have power.”

“You sure, sounds like the onboard computer fuse might be blown.”

So, I trudged out and took off the fuse cover.  Now, I don’t really believe in fate or signs from God, but it was weird, in the list of about 40 fuses, the first one my eyes fell on was Computer. “Nah, can’t be.”

I pulled it, and sure enough, it was burned out. I pulled it and replaced it. Got in the car and thought, “it can’t be that easy, can it?” A turn of the key and the next thing I knew, the 6 cylinders were purring.

All that work and frustration because I had overlooked the basics.

This is far from the first time I’ve overlooked the basics. And I bet you’ve done the same thing. I have a theory about why we do this, and it is in part because the basics ARE so fundamental that we assume it has to be something else. In my model train example, dirty track and loose wires, especially in an ad-hoc setup are arguably a more common issue than forgetting to plug in the transformer. In my BMW case, because literally everything else worked, I assumed the power was getting to the computer. And honestly, even now, thinking about it, I’m surprised the dash light startup didn’t change at all because of a lack of computer.

I’ve seen this in databases and elsewhere. I was recently trying to do a quick restore of a database from one machine to another and the obvious wasn’t working. It took me a bit to remember the client’s new security setup prevented this specific case for these two machines. Once I remembered that, the problem and subsequent solution were obvious.

This in part goes back to why I like using a rubber-duck at times. It can force you to review your assumptions and check the basics.

Having a problem? Employ the Charlie M. rule and check the basics.

 

JOBS THAT BEAT THE CARING OUT OF YOU

Let me start by saying this is NOT an April Fool’s Joke. This is a true story.

I do lay the ‘blame’ for this post squarely two members of my #SQLFamily: first on the heels of Grant Fritchey and his post by he same name. He in turn lays blame on Jen McCown’s post by the same name.

I mention elsewhere in my blog I prefer to be intelligiently lazy, so rather than retype, I’ll post the content from a Quora answer I wrote.  Technically I was just a consultant, and after twice getting a late check I made it clear to them that if they stopped paying me on time, I would stop working.  Apparently they liked me enough that a quick call to the CFO would get me a check cut that day.

So with that:

Let me give you an example of a client I once had. When I started with them, people loved working there and they were expanding and successful. So successful the company got bought.

Then… things changed.

Sales people were finding their expense checks weren’t getting paid (more on that later). Did you know, even if you try to explain to the credit card company that it’s a “company card” if it’s in your name and the company doesn’t pay it, you’ll ruin your credit score? Yes, it’s pretty difficult to be a sales person who can’t travel because no one will give you a credit card any more!

Then, to cut costs, an office move was proposed. Quite frankly, had I not been involved as their IT guy, it would have been a disaster for a variety of reasons. Fortunately for them, besides my IT skills, I could read blueprints. It was quite obvious to me that 2 outlets would not serve an office of 20–25 people with computers and printers. It took me nearly kidnapping the CFO on a day he visited and dragging him to the office to make clear how much more work the office needed. They simply assumed, “oh, it’ll have enough power.”

Meanwhile the previous owner had started a new company (in a completely different industry) and was growing and expanding at a furious rate. Also, my wife was a recruiter at another local company (in a different industry also). The only thing all three of these companies had in common was they all were software related, but the fields they served were completely different.

At one point, the top sales person from the failing company left to go get a job a with the new company. Within days the former company sent a cease and desist letter to the new company insisting they stop poaching employees and if they continued, they’d sue the owner for violating the non-compete clause. Now, keep in mind the owner was very much NOT approaching employees of the old company, but even if he were, the non-compete only applied if he had founded a new company in the same industry. he hadn’t. We had a good laugh at the old company.

Now, meanwhile, my wife, while not exactly poaching, knew that almost any offer she made would be accepted since morale was so bad at the old company.

Then… this happened. I was there for the meeting and sat in on it. It’s the closest I’ve come to “beatings will continue until morale improves” ever.

The CFO and CEO came into town for an all-hands meeting. Their goal was to address, among other things, the late employee expense checks issue.

I will say, they had some pretty looking slides. The slides showed things like cash-flow, moving towards profitability and some other items. But the message was quite clear, “We will continue to pay YOUR expense checks as late as possible because it helps our cash flow. And you should be grateful for this.” They very much could NOT understand why employees were furious that their expenses were basically being used as no-interest loans by the company. The rate of exits accelerated after that.

What had been a thriving company became a dying, decaying shell of a company in under a year because of the management.

One Postscript:

One of the developers who left the old company ended up at the new company. He submitted his expense check. He was reasonable, he knew it would probably hit his next pay cycle. He was OK with that. I still recall the look on his face when later that day someone from finance walked in with his expense check. They were under no obligation to turn it around that fast and he certainly wasn’t expecting it. But they did so. They “bought” his loyalty that day by a simple gesture.

So, if people are leaving, trying to force them to stay will backfire. Figure out what you’re doing wrong and fix it.