About Greg Moore

Founder and owner of Green Mountain Software, a consulting firm based in the Capital District of New York focusing on SQL Server. Lately I've been doing as much programming as I have DBA work so am learning a lot more about C# and VB.Net than I knew a couple of years ago. When I'm not in front of a computer or with my family I'm often out caving or teaching cave rescue skills.

Call 911, If You Can

Also known as “things have changed”

For one my of clients I monitor and maintain some of the jobs that run on their various servers. One of them had started to fail about two weeks ago. The goal of the job was basically to download a file from one server, transfer it to another and upload it.  Easy-peasy. However, sometimes the job fails because there’s no file to transfer (which really shouldn’t be a failure, but just a warning).  So, despite the fact that it had failed multiple days in a row, I hadn’t looked at it. And of course no one was complaining (though that’s not always a good reason to ignore a job failure!)

So yesterday I took a look and realized the error message was in fact incorrect. It wasn’t failing because of a lack of a new file, but because it could no longer log into the primary server. A quick test showed the password had been changed. This didn’t really surprise me as this client is going through and updating a number of accounts and passwords. This was simply obviously one and we had missed this one. (Yes, this is where better documentation would obviously be a good idea.  We’re working on that.)

So, I figured the fix would be easy, simply email the right person, get the new password and update the process.  I also was taking the time to update the script to that the password would be encrypted moving forward, right now it’s in plain text and to give the correct error in the event of login in failure.

Well, the person who should have the password wasn’t even aware of this process. As we exchanged emails, and the lead developer chimed in, the conclusion was that this process probably shouldn’t be using this account, and that perhaps even then, this process may no longer be necessary.

So, now my job is to track down the person who did or does rely on this process, find out if they still are and then finish updating the password.  Of course if they’re not, we’ll stop this process. In some ways that’s preferable since it’s one less place to worry about a password and one less place to maintain.

Now, the above details are somewhat specific to this particular job, but, I’m sure all of us have found a job running on a server someplace and wondered, “What is this doing?” Sometimes we find out it’s still important. Sometimes we discover that it’s no longer necessary. In a perfect world, our documentation would always be up to date and our procedures would be such that we immediately remove unnecessary jobs.

But the real world is far messier unfortunately.

(and since the full photo got cropped in the header, here it is again)

Call 911. If you can

Apparently not only can guest rooms can not be called from this phone

And as a reminder, if you enjoy my posts, please make sure to subscribe.

 

Tighter than I Remember

I’ve been reading a book off and on for about a year now called, Being Wrong : Adventures in the Margin of Error. It ties into my interest in meta-cognition and how we think.

We build a model in our head of how the world is. Often times it’s accurate, but often times it’s wrong. In my model of the world, the Sun rises in the East. This model is accurate and corresponds with the model most of us have. But I can easily construct related models that are wrong. Looking back I can remember that it was sunny on a particular day, only to find if I check the records, it was raining. My memory is wrong, but it created the model I had for that day. I can also create a model going forward. Last night, going to bed, the world I was going to wake up to was sunny. This was based on the weather forecast and the fact that the Sun rises in the East. Well, the Sun may have risen today, but it’s not sunny. There’s far too many clouds. So that model was wrong.

This all comes to mind because of a caving trip I did this weekend. There’s a fine little cave near here called Ella Armstrong. It’s really not much of a cave, perhaps 250′ of walkable passage total. But I’ve always liked it even though I hadn’t been there in 15-20 years. It’s got a nice vertical drop at the entrance that’s great for beginners. And it has an easy walk-in entrance.  As a result, I had been considering using it for some cave rescue training next year.

So you can imagine my surprise when two of my fellow cavers who had been there just a few weeks ago said I was in fact wrong and it’s not a walk-in entrance and that in fact it’s rather tight. I was surprised. I didn’t really doubt them, but figured perhaps we had a different definition of what we considered a tight entrance.

Now, let me take a little detour here and mention that almost every caver will at some point joke about how a particular cave has gotten smaller over the years. It’s a little lie we tell ourselves to account for the fact that many of us have put on a few pounds since we first started caving and some passages are in fact harder to get through.  The truth is, rarely due caves get smaller, though sometimes passages sometimes do get larger. There’s a cave in Vermont where I’m pretty sure fitting through 2 of the 3 tightest spots is probably close to impossible for me now. And I know the cave didn’t change sizes, so I’ll have to admit I have.

But back to Ella Armstrong. If you had asked me to describe the entrance prior to this weekend I’d have described it as about 12′ tall at its tallest and 3′-4′ wide most of the way to the top of the drop.

Well, as Saturday proved, the model in my head was only half right. The map lists the entrance as 8′. I’ll call that close enough to 12′ since it simply means there’s no chance to hit my head on the ceiling.

On the other hand, most of the entrance from the surface to the top of the drop is NOT 3′-4′ wide. It probably averages 18″ at most and in parts is just wide enough for me to work my chest through with some contortions. My first reaction when I got to the entrance was, “perhaps some rocks have fallen in and narrowed this from what I remember.” But, after I started to scramble beyond some of the rocks on the surface, I got to the bedrock, which I know hadn’t moved and realized that my model was dramatically wrong. It was tight enough that at one point I considered calling off our trip. It wasn’t that I didn’t think I couldn’t make it further down and past a particular spot; gravity would see to that. It was more that I was afraid I might not get back UP past that spot. Finally after some mental gymnastics I figured a way I could not only get down past the choke point, but felt comfortable thinking I could get back up past it. As I’m writing this from the comfort and safety of my home, you can see my mental gymnastics worked.  Actually, it turned out the crux move on the way out was a bit further up that wasn’t nearly as tight, but had a short shelf about 3′ up in a spot that wasn’t great on foot or handholds. On the way in, that spot didn’t appear to be an issue at all. So again, in the narrow space of about 20 minutes my mental model failed me.

The truth is, we make mental models all the time. It’s how we operate in the world. But sometimes those models are wrong. Sometimes in a positive way, the tight spot getting out wasn’t as bad as I thought, or in a negative way, the spot I thought would be easy, was in fact the hardest sport. But regardless of the errors in many of our models, we generally navigate the world in a successful manner. Being wrong isn’t necessarily the end of the world.

Unfortunately though, as much as I love this cave, it really is too tight to be practical for cave rescue exercises. Which is a shame, because it is a great little cave. And if you’re ever in the area and want to check it out, let me know. I’d love to take you. But I’ll warn you, it’s a bit tight in spots!

Sincerity

There’s an old sales joke about sincerity: If you can fake it, you’ve got it made and can sell anything. I was reminded of this at the SQL Saturday Virginia event. The location was the ECPI campus in Virginia Beach. From talking to Monica Rathbun, the event organizer they were amazing (and more about that in a bit).

But… first a bit of a laugh (at least to me)

Best decision ever made

Best decision they ever made!

I mean I suppose it’s possible both of them had the exact same thought and expressed it in the exact same way, but I think I’d have posted the two posters far enough apart that it didn’t seem so obvious these quotes were probably made up.  It didn’t really come across as sincere. I was amused.

That said, as I understand it, ECPI donated the space for the weekend, setup the tables, took them down and basically did all the site-work that normally SQL Saturday organizers have to do. So kudos for them. Also, at least one professor sent his students down to check us out and gave them extra credit if they went to any of the sessions.

So, in that sense, I will say, I think the folks at ECPI were very sincere in supporting the event and really appreciated it.

But more so, I appreciated what appeared in the speaker room later in the day:

Good eats!

Speaker afternoon snacks

and

More speaker snacks

More speaker snacks

I spoke to the chef who brought them in and apparently ECPI has a culinary school and this was the day of their practicals.

I’ve seen a number of places where food is provided and generally the food staff do an adequate job.  One thing I noticed here was how professional the catering staff were. They wore the typical white jacket of chefs. But, despite being basically a cafeteria across the hall (and the culinary school which was apparently in another building), they acted like this was the Ritz-Carlton and we were buying $75 steaks. Their professionalism and, yes, you knew I was getting here, sincerity in doing their job stood out.

I appreciate it when someone does their job well and sincerely and isn’t just putting in the time.  I recognize we can’t always do that and we have our bad days, but in general, if one can be sincere about their work, I think they should be. It can be obvious when someone doesn’t treat their job seriously or sincerely.

All in all, a great SQL Saturday and I say that sincerely.

 

 

Rolling Back

I’ve had a number of thoughts rolling around my head lately and they’re somewhat related so here goes.

I’m going to start with a depressing headline and URL: Civilization will end in 2050. Ok, perhaps that’s more of an alarmist headline than anything else. I posted this elsewhere, and some folks correctly pointed out that this could have been written by Thomas Malthus over two centuries ago.  And it’s true; prognosticators have predicted the fall of humans since probably before recorded history.  How many of us recall in the last century alone predictions of world-wide famine, a world-wide ice age, comets smashing into us and other horrible events.

Some were prevented, such with the Green Revolution. Some were corrected with better data or a better understanding of the data. Some where just… nuts.  That said, some predictions can be made with 100% certainty, even if the bounds on the actual circumstances are fuzzy. I’ve been giving this particular nugget a lot of thought.  Since I’ve passed that magical (at least to humans) mark of a half-century, I’ve given more thought to what I call my expiration date. It’s a fact. At some point I will cease all biological functions and will be dead.  I can’t escape that. Every day the odds increase a very tiny percentage.  So, I am predicting my own death, most likely sometime in the next 1/2 century. There’s a very slight possibility it could be tomorrow, and a much greater chance it could be 40 years from now, and based on current medicine an absolute certainty 80 years from now.  You’ll note I hedged that last one. It’s quite possible in the next few decades we figure out how to extend the human lifespan by decades if not centuries.  But I’m not counting on it. And even then, there are no guarantees I’m not hit by a driver while biking or some other catastrophic event.

That said, right now I’d say the odds are decent I’ll be alive in 2050. And almost certainly my kids will be (and very likely I’ll have grandkids by then).  So, am I worried that the world will end in 2050? Yes and no. For one thing, there’s time between now and then for a lot to happen. But, as we get closer, it’s going to be harder and harder to forestall the impacts of higher levels of CO2 in the atmosphere and oceans. Yes, to be clear, I believe the evidence supports that humans are greatly impacting the climate via CO2 and other emissions.

But I’m also optimistic that we’ll work harder, especially as things get worse, to stave off the most pessimistic scenarios. But it won’t be easy and the longer we wait, the harder it will be.

While anecdotal, I’ve already seen the impacts in my life, earlier springs, caves that once held ice year round no longer do, less snow in the winters, etc. It worries me. But I also still have some hope.

That said, an analogy that’s been rolling around in my head that isn’t 100% perfect but fits me as a DBA.

Many of us in the SQL server community have started a long running transaction, only to realize we’ve brought the server to a halt. In other words, no other user can access their data until our transaction is complete. We can abort the transaction, but if we wait too long, that can be a far worse solution than trying to stop it early on. For the non geeks, in other words, if our transaction is going to update 1 billion records and we realize 5 minutes in that it will take 10 hours, we can abort it and it should take about 5 minutes to rollback to the original state. This means we’ve only brought the server to a halt for about 10 minutes. However, if we ignore the problem, and keep pretending it’ll go away and we wait say 9 hours and then finally decide, “oh what the heck, let’s try to fix the problem now” it might take another 9 hours to rollback. In other words, by waiting to long to resolve the problem, instead of being 10 minutes, it can be 1080 minutes.

In other words, the longer you wait, the harder it can be to recover. I think that’s where we’ll be at by 2050. Whether or not I’m still alive (though I plan to be!)

Being Observant

I was busy last week teaching cave rescue at the annual National Cave Rescue Commission Weeklong training seminar, hence my lack of a post last week.

As always, I had a great time helping to teach a class of 19 students. Our teaching includes classroom time as well as time in the field. We use both caves (obviously) and cliffs for out in the field instruction. The cliffs give us an opportunity to focus on rigging and lets everyone see what’s going on; mostly. In a cave, often it’s too dark and small to let everyone have a good few.  But, you may notice that I said mostly above.

Let me interrupt with a quick video.  Can you count the number of times the players dressed in white pass the basketball? If you got 15, congrats.

Now back to the class being on the cliffs. As part of our exercises, we usually do several iterations of the same thing, such as lowering a patient and then raising them up. Each time we may change a detail or two, such as where they can rig, what haul system to use, etc. The idea is that repetition helps them learn the skill. Often though, at the last iteration we toss in a wrinkle: they’re not allowed to speak, at all. Generally this means, they have to start with a pile of rope and hardware on the ground and without saying a word, rig a safe system to lower and raise a patient.  Generally they use a variety of hand signals and various body motions. While at times it can be quite entertaining, it can also be instructional. It’s often the faster iteration of the day, in part because they’ve honed their skills but also due in a large part because there’s no ideal chit-chat.

The point of the exercise is multi-fold. Partly it’s a great challenge for them that they end up really enjoying. It also illustrates that communication is still possible when you can’t verbalize things. This is actually often common in caves where rushing water or echos can make vocalizations useless. It finally shows that they’ve come together as a team and can work effectively.

That said, sometimes they can miss details. As instructors we ensure though that no safety issues are missed. We will stop the exercise if we see something unsafe and help them make it safe.  Part of this includes an instructor on a separate rope to hang over the edge so they can watch the patient (generally another instructor) go down and come back up. This instructor watches the entire process for safety reasons.

This time around, we decided to add a bit of a twist to the exercise.  Generally the safety instructor stays at the top. This time however, he rappelled over the edge, all the way to the bottom. Nothing overly unusual about that.

However, once he was at the bottom, along with the patient, who happened to be me, I very quickly detached the ropes connected to my harness and connected them to his harness. I then moved over to the line he had been on and started to ascend as the students began to haul him up.

They eventually got him to the top and congratulated themselves on a job well done. Other than the student at the edge of the cliff who had watched the process, none of them appeared to notice that they had lowered me and raised him.

This actually isn’t totally surprising. We like to think of the human brain as a computer, but in many ways it’s not. Or at least it’s not a digital one with perfect recall. We actually ignore a lot of “noise” in order to focus on what’s important at the time. The fact that the person they hauled up was different from the person they had lowered counts as noise in this case because it’s so improbable that the brain won’t bother processing it, lest it take away from more important tasks, like ensure the ropes were handled safely.

And for those of who you didn’t watch the video, the answer is 15, but that’s not what’s important. Go watch it.

Today’s takeaway, our brains are fuzzy and work “well enough” most of the time, but can miss details. And generally, that’s ok.

Marshmallows Part II

I’ll have to admit, I can rarely tell in advance when one of my posts will hit all the buttons and generate views and when it’ll fall flat. But as I don’t always write for my audience, sometimes I write for my own reasons, I can live with that.

So, how to follow-up on a post that didn’t receive many views, write a follow-up post. You can call me a slow learner.

Actually, it’s about learning. Last time I wrote about my microwave and doing a quick experiment with marshmallows to prove it was really dead.  After 2 days without a microwave it was time to get a new one. Of course I couldn’t get what I wanted because the space it had to fit into was limited in size.  That could have been resolved, but would have meant redoing the cabinet space it had to fit into. And if I were going to redo the cabinet space there, I might as well redo the rest of the cabinets. And if I’m going to redo the cabinets, I really need to redo the counters. And very quickly a replacement $100 microwave I can get in an hour would become a 3-week $10,000 kitchen remodel. I opted for the $100 microwave over the one I really wanted.

And the results are shown at the top of post (and below in case the top doesn’t appear)

melted marshmallow picture

10 seconds of marshmallows in the microwave

It’s quite interesting to me. The best heating was beyond the area of the rotating plate.  But this also shows the value of the rotating plate since if there’s a few sports, if I put a something to heat and everything was stationary, it would take forever to heat since there’s little to no microwave energy there. (This can get complex because of the size of the wave and the height of the material, etc.)

Now, I’d have done more experiments, but it seems a certain someone in the house enjoys marshmallows more than I do and had eaten a bunch and this was all I had.

But, I have a working microwave and I’ve proven how important the rotating plate can be (not that I had much doubt).

And that’s science to me; doing experiments and learning.

Oh and about the SQL query I was updating. It’s going into production this week hopefully. I was about to eek out about a 10-20% improvement. Beyond that, not much I could do because it really ends up scanning an entire table, on purpose.  Only so much you can do there.

One last thing: there may not be a post next week because I’ll be teaching at the NCRC weeklong cave training class in Indiana and will have limited internet and time.

Marshmallows

Though I attended RPI, which is generally considered an engineering school, my degree is a BS in Computer Science. I say that because I consider myself more of a scientist than an engineer at times. And honestly, we all start out as scientists, but many of us lose that along the way.

Anyone who has had a small child has observed a scientist in action. No, they’re not in a lab full of test tubes and beakers and flasks giving off noxious smells. But they are in the biggest lab there is, the world. They also don’t necessarily realize it. Nor do parents. But every time they drop a Cheerio, they’re testing gravity.  Fortunately (or unfortunately depending on your point of view) so far every time they’ve managed to prove that gravity works. This is the most obvious example, but when you stop to think about it, much of the first few years of life is all about experimenting. Most of the time it goes well, but sometimes, as a burnt hand will attest, the experiment has a less than ideal outcome.

And it’s the fear of burned hands that leads to parents to utter that common  refrain, “Don’t touch that!” or the variation “Don’t do that!”.  Soon, over time, our experimentation starts to get reined in until we do very little of it. This can be inhibiting.

Years ago I used to teach an “Introduction to Windows” adult education class. It was I believe a 6 week class and I taught several over the course of a couple of years. It didn’t take me long to realize the biggest constraint on the students ability to succeed in the class was that they had internalized “Don’t do that, you might break something.” Once I realized that, half my teaching pedagogy simply became, “Touch that, you won’t break it, and if you do, it’s not a big deal, and if it is, we’ll fix it anyway.” Seriously, more than anything else, I had to encourage most of my students to experiment with the computer.

More recently I realized I had stopped doing as many experiments in my life as I should be doing. About 1.5 weeks ago I attended a Wilderness Medicine Conference a friend of mine had told me about. At the end of the very wet, cold, rainy day, a bunch of us went outside and tried to start a fire. Starting a fire, let alone in such conditions was something most of the students had never done. I had, but not in years. With some effort, and experimentation, including using the outside box of a single serving size package of Fruit Loops, we finally managed to get the fire going.

But this got me thinking. When I go hiking, I carry a tiny ziplock back in my jacket with some firestarting materials. They’re there in case of an emergency. But, the thing is, I had never actually tried them and realized if I didn’t know how well they worked in practice, I couldn’t rely on them in emergency. So, I went outside, and started a fire. And I learned that yes, my materials ARE adequate, but the dryer lint needed to be pulled apart more than I realized. I tried again later in the week, and added the use of a toilet paper roll to form sort of a chimney so the starting fire would draft better. This, and the better pulling of the lint worked even better and a single match was sufficient this time.  This gave me more confidence that in an emergency, in less than ideal conditions I could get an actual fire going.

But, I wasn’t done! Our microwave broke this weekend. But, before I wrote it off, I wanted to make sure it wasn’t a fluke or something else. So, in this case I decided to get a bag of marshmallows and lay them out inside the microwave to see if I was getting ANY energy out of the magnatron. Turns out, nope, nada, nothing. So, today or tomorrow I will be buying a new microwave. But, it was a fun, and later tasty experiment.

Without delving deep into the scientific method here, I’ll say at a simple level, science is about having a hypothesis and testing it. The testing it is important.

To bring this back to SQL. First, you have a hypothesis that your backups will work. Have you tested that hypothesis? If not, do so immediately. Even if they do, you might learn something now that will be important when you have to do it for real. Perhaps you learn the volume your backups are on only has write access. Or perhaps you learn you need to retrieve your encryption keys and the person who controls access to them is on vacation. Or perhaps your RPO is 4 hours and the restore takes 6 hours.  So, experiment.

query plan

Capture of a random query plan

Recently for one client I’ve spent some time experimenting with various changes to help improve the performance of some queries. Not everything I tried worked, but some things did. So, again experiment.

I’m curious what recent experiments you may have done, SQL or otherwise. What were their outcomes?