Snow Days

“I sometimes hear a moment before sleep, that I can never remember whether it snowed for six days and six nights when I was twelve or whether it snowed for twelve days and twelve nights when I was six.” – A Child’s Christmas in Wales by Dylan Thomas

As I’m writing this there is snow gently falling from the sky and the ground is covered.

I woke up this morning to the sound of a plow scraping the roads clear.  I got up to check the school closings list, expecting at least a 2 hour delay. Somewhat surprisingly there was none.

But it got me thinking about how the same event can be perceived differently by different people.

As a young kid, many of us loved the idea of a snow day.  We hoped we’d wake up to the soft hush a blanket of snow causes, broken only by the occasional scrape of the snow plow. Perhaps we might hear the sound of wheels spinning as a car tried to gain traction to keep on its way. Some even created rituals, such turning their pjs inside out, or sleeping with a spoon under our pillows. (For the record I actually never even heard of any of these until I was an adult). A snow day meant a day of fun in the snow: building a snowman, or better a snow fort and having snow ball fights.  I recall one particularly expansive snow fort friends and I built in a snow bank in the center of Falls Village where we grew up. It had a main chamber from which we could survey our domain and at least two side tunnels we could craw through, leading to smaller “towers” that could fit one of us, to provide flanking fire for anyone foolish enough to try an assault on the main chamber.

Sometimes we’d even play the hero and after one blizzard at least, a friend and I went through town, uncovering buried cars, just in case anyone was trapped. Fortunately no one was. Of course we also then had to at one point dodge a snowplow by scrambling through a 5′ embankment of snow created by previous plows.

As we got older, we may have given up on the rituals and built fewer snowmen, but we still enjoyed our snow days. It meant a break from school, perhaps a chance to catch up on homework. But it also often meant chores, the need to shovel the walk, or worse the driveway.

Then we got older still and now we didn’t get days off. We were told, “the office is still open. Please drive safely.” Now those spinning tires we heard as a child were us, trying to keep straight, and on the road, in order to get to work. Those snowplows we hoped to hear as a child were both a boon and a bane. They helped clear the roads, but also seemed to be in the way.

If our children were young enough, suddenly a day off from school for them, became a burden for us as we struggled to find a sitter or some form of daycare.

We no longer looked forward to forecasts of snow. We dreaded them.  We started our own rituals, some actually more effective than what we practiced as a child. We’d pre-salt the walkway. We’d make sure we had a snowbrush inside the house ready to go so we could clear off the car before opening the door.

It was the same event, but a completely different perspective.  I think I preferred the childhood perspective.

And the irony is not lost on me that my job now actually permits me to sit at home, avoiding the drive, and to write about the snow.

For me, even when I have to drive in it, I actually love the snow and snow days.

For you, I hope you get the day you want, young or old, snow or not.



When things aren’t the same

A recent thread on the Nanog mailing list regarding the outages this past weekend reminded me of one of my war stories that I thought I’d relate.

About 7 years ago, I was involved in a project to help a different division of the company upgrade and replace their cluster and SAN with new hardware.  I was in a fairly hands-off mode, but towards the end was more involved.  We had outsourced the hardware and software installation to another company, but we were about to take over and move the production system to it.

I tend to be a bit paranoid when it comes to changes.  Changes can introduce new, unknown points of failure.  There’s a reason for the saying that “better the devil you know.”

Anyway, before we were about to go live, I asked, “Has anyone tried a remote reboot of the box, just to make sure it’ll failover correctly and come up?”

“Oh sure, we did that when we were in the datacenter with the vendor a couple of weeks ago.”

Now, there had been several changes to the system since then, including us adding more components of our software.  So I did a quick poll and asked if everyone was comfortable with moving forward without testing it.  The answer was in the positive.

Still, as I said, I’m a bit paranoid.  So I basically asked, “Ok, that’s nice, but do me a favor, reboot the active node anyway, just to see what happens.”

So they did.  And we waited.  And we waited.  And we waited.  As I recall, the failover DID work perfectly.  But the rebooted node never came up.  So we waited.  And waited.  Now, unfortunately due to budgetary reasons, we didn’t have IP-KVM setup.  We had to wait until we could get to the remote datacenter to see what was going on.

Meanwhile, the vendor that did the install was called and they assured us it was nothing they had done. It had to be a hardware issue, since they had installed things perfectly.

That’s your typical vendor response by the way.

Finally we get someone out to the datacenter.  What do they see, but the machine waiting to mount a network share to get some files.  This puzzled us, since we had no network shares for this purpose.  We pondered it a bit.  Then it dawned on me that the only thing different between our reboot and the earlier one, was the vendor was in the datacenter.

So, we made the necessary changes to not try to mount network files, tested the reboot a few times and all was well.

One of my team called the vendor and asked, “hey, when you do an install, does your tech mount stuff off of their laptop?”  “Oh sure, that’s how we get some of the critical files where we want them, why?”

So, one little change, having the vendor there (with their laptop on the network) and not having the vendor their made all the difference.

If you’re going to test, test in real-world scenarios as much as you can.  Make sure nothing extra is plugged in, even if you think it won’t make a difference.

If you can, test it just the way you would expect it to fail in the real world.  If that means you have redundant power supplies, test it by tripping the breaker on one leg.  If you find yourself saying, “Oh no, that’s too risky” guess what, you’re not nearly as protected as you think you are.  If you can suggest doing the test with confidence you’re 1/2 way to success.  If you actually DO the test and things work, you’re all the way.

In conclusion, one seemingly innocuous change can make a huge difference.