Author Archives: Greg Moore

Unknown's avatar

About Greg Moore

Founder and owner of Green Mountain Software, a consulting firm based in the Capital District of New York focusing on SQL Server. Formerly, a consulting DBA ("and other duties as assigned") by day, and sometimes night, and caver by night (and sometimes day). Now, a PA student working to add PA-C after my name so I can work as a Physician Assistant. When I'm not in front of a computer or with my family I'm often out hiking, biking, caving or teaching cave rescue skills.

QuiCR’s latest product

Posted on by
Reply

I mentioned in my latest post I was working on a new project, one that valued simplicity over complexity.  I can now talk about it a bit more.

I had approached a local cab company about the QuiCR product.  Unfortunately, given who his largest market demographic was, (most of his fares do not have cell phones, let alone smart phones) we decided it wasn’t a great fit.

But, as I mentioned, he had made an off-hand comment about something he would like: namely an ability to allow a smaller demographic of his, the local college students, the ability to send a text to his dispatcher and request a cab.

This is one of those design ideas that’s both deceptively simple and complex at the same time.  It’s simple because “Receive text, display text, allow a response” about describes the problem.

Now, the simplest solution obviously would be a to give the dispatcher a cell phone with texting capabilities.  That would also be the wrong answer.  For one thing, his dispatchers work at a frantic pace and time is off the essence.  While some folks may be able to whip off text messages using “text-speak” in seconds, his staff isn’t among those with fingers that nimble.  

It also doesn’t provide for easy reviewing of messages and threads and the like.

So, the trick was coming up with a computer interface that was simple enough that it could be adopted with only a few minutes of training and that wouldn’t interfere with their current manual dispatch system.

The keyword there is manual. Yes, there are systems out there with all sorts of bells and whistles that can integrate with GPS, credit card systems, IVR and much more.  Those systems also costs a LOT of money.  And in at least one case, a vendor was suggesting that to adopt it, he hire another dispatcher to handle the increased load.  Note the load wasn’t necessarily from increased business, but simply from the complexity of the system.  Now, don’t get me wrong, in a large city where you have dozens of cabs, such a system is the right approach and scales well.  But, it doesn’t scale very well to smaller companies.

His dispatchers use a very manual system.  And it works. Hopefully my new “text-dispatcher” will integrate well with the current system and generate some new business for him.

Sometimes, simpler is better, but harder

 

Timers

Posted on by
Reply

There’s a photo I recall of the Space Shuttle flight deck looking out over the cargo bay.

What’s most interesting about it is the number of kitchen timers velcroed to various surfaces.  They were definitely afterthoughts.

After a few flights NASA started to realize that their multi-billion orbiter was short a few $1.50 timers on the flight deck.  They were used for things like timing rendezvous maneuvers, arm movements and more.  It was such a simple  need that was evidently overlooked early on.  Or perhaps not so much overlooked as much as no one foresaw the usefulness.

I was thinking of this tonight as I was cleaning up in the kitchen and realizing how useful timers are so I don’t forget things. In this case, putting the wok on the hot stove to boil off some water.  Something I could easily walk away from and realize an hour later I had forgotten about. Sure a pretty mundane use, but a very important one.

I’m also reminded of it as I’ve submitted a proposal to a local company to add something new to their business.  The idea came from an off-hand comment the business owner had made.  More importantly though, he liked the prototype because I listened to his other comments.  This had to integrate into an existing manual system as easily as possible.  He had others come to his business and suggest multi-thousand dollar solutions that would require he hire MORE staff and send them all to training.  That was not a winning business solution.  If my proposal is recommend, it will involve a laptop on the desk and at most 1-3 mouse clicks to do what he wants his team to do.  Time in training… 5 minutes. Cost.. under $1K upfront (and under $1K monthly).  Possible impact on his business >$1K a month.

That’s hopefully a winning combination.  I’ll know tomorrow if it works.

But sometimes the simplest approach is the best approach.

 

When things aren’t the same

Posted on by
1

A recent thread on the Nanog mailing list regarding the outages this past weekend reminded me of one of my war stories that I thought I’d relate.

About 7 years ago, I was involved in a project to help a different division of the company upgrade and replace their cluster and SAN with new hardware.  I was in a fairly hands-off mode, but towards the end was more involved.  We had outsourced the hardware and software installation to another company, but we were about to take over and move the production system to it.

I tend to be a bit paranoid when it comes to changes.  Changes can introduce new, unknown points of failure.  There’s a reason for the saying that “better the devil you know.”

Anyway, before we were about to go live, I asked, “Has anyone tried a remote reboot of the box, just to make sure it’ll failover correctly and come up?”

“Oh sure, we did that when we were in the datacenter with the vendor a couple of weeks ago.”

Now, there had been several changes to the system since then, including us adding more components of our software.  So I did a quick poll and asked if everyone was comfortable with moving forward without testing it.  The answer was in the positive.

Still, as I said, I’m a bit paranoid.  So I basically asked, “Ok, that’s nice, but do me a favor, reboot the active node anyway, just to see what happens.”

So they did.  And we waited.  And we waited.  And we waited.  As I recall, the failover DID work perfectly.  But the rebooted node never came up.  So we waited.  And waited.  Now, unfortunately due to budgetary reasons, we didn’t have IP-KVM setup.  We had to wait until we could get to the remote datacenter to see what was going on.

Meanwhile, the vendor that did the install was called and they assured us it was nothing they had done. It had to be a hardware issue, since they had installed things perfectly.

That’s your typical vendor response by the way.

Finally we get someone out to the datacenter.  What do they see, but the machine waiting to mount a network share to get some files.  This puzzled us, since we had no network shares for this purpose.  We pondered it a bit.  Then it dawned on me that the only thing different between our reboot and the earlier one, was the vendor was in the datacenter.

So, we made the necessary changes to not try to mount network files, tested the reboot a few times and all was well.

One of my team called the vendor and asked, “hey, when you do an install, does your tech mount stuff off of their laptop?”  “Oh sure, that’s how we get some of the critical files where we want them, why?”

So, one little change, having the vendor there (with their laptop on the network) and not having the vendor their made all the difference.

If you’re going to test, test in real-world scenarios as much as you can.  Make sure nothing extra is plugged in, even if you think it won’t make a difference.

If you can, test it just the way you would expect it to fail in the real world.  If that means you have redundant power supplies, test it by tripping the breaker on one leg.  If you find yourself saying, “Oh no, that’s too risky” guess what, you’re not nearly as protected as you think you are.  If you can suggest doing the test with confidence you’re 1/2 way to success.  If you actually DO the test and things work, you’re all the way.

In conclusion, one seemingly innocuous change can make a huge difference.

 

Change is good

Posted on by
Reply

So, for the “fun” of it, I added some dns names to my home network (namely the cablemodem and the router).  Now, it’s not like I really NEEDED to do so. The honest truth is IPv4 network addresses are rather simple to remember. So, I could just type in the IPv4 address of the device I want to reference and do fine.

So why bother?

Well, a former colleague of mine convinced me that it was time to change.  You see, IPv6 is coming.  I won’t bore anyone with the details, but let’s just say that I’m not about to try to remember the IPv6 addresses of any of my devices.  At first, the Luddite in me rebelled at the idea that I could no longer just refer to devices by IP address.  I attempted a few feeble attempts at arguments (hey, it’s easy to remember the IP address of some public DNS servers for example, and that’s one of the few times in an IPv4 network it really helps to have memorized some IP addresses.)

But the truth is, over time, I saw his point.  IPv4 was the old way.  IPv6, for a variety of reasons is the future.  My desire to be able to say I could remember the IPv4 address of my router was simply stubbornness.  The truth is, there really isn’t any real benefit to it.  The fact that I could (and did) reference my network printers by IP address wasn’t really helpful.  In fact, it meant that when I wanted to move it to a different address, I had to go to individual machines and make changes.  Now, I had  a small network, and only changed the printer address once in the past 5 years.  So it wasn’t a huge burden.

But now that I’ve moved to using hostnames, even for stuff I used to use IP addresses for, my limited memory can be used for more useful things.  And when I do reconfigure my network (say add another printer, or expand it in other ways) a simple DNS change and I’m all done.  I don’t have to go from machine to machine to make changes.

So, the moral of the story is, just because IPv4 made something EASY to do, it didn’t make it the right thing to do. IPv6 forced me to reconsider my thinking and in the end change for the better.

Sometimes, being forced to change can make you a better person.

SQL Server User Group

Posted on by
Reply

Not much of a post tonight.  I didn’t realize it had been over two months since I last posted.  I’m sure all my faithful readers (all 1 or 2 of them?) have been holding their breath.

Anyway, tonight went to the local SQL Server Users Group meeting.  They had a remote demonstration on query tuning, one of my favorite topics.

In fact performance tuning in general has been a topic I’ve often enjoyed.

First rule: There is always a performance bottleneck.  This is one of the first rules I recall reading.  At first people object, “but the system is fast enough.”  That may be true.  BUT, there is still limiting it from being better.  Of course you might not be able to fix that limit.  But more importantly, it may not matter.

Second rule: It may not matter.  If your query is already running in subsecond times, it may not be worth spending any time on optimizing it any further.  Or, if your query takes 1 hour to run, but runs at night when nothing else is running, it may not matter.

Third rule: Optimize only what you need to.  A classic example of this I’ve seen is finding reports that run overnight and slow down other processes.  You start to optimize it and then think to ask, “is anyone still using this report?”  You find out the report is no longer being used.  Now you can achieve the holy grail (and perhaps the only exception to rule 1): infinite optimization.  Delete the report and suddenly you have infinitely optimized it.

BTW, these rules don’t apply just to SQL Server.

Optimize your life and enjoy more of your time doing things you enjoy.

The Hunger Games

Posted on by
3

I’ll admit it, I’m a sucker for “fantasy” worlds. I don’t necessarily mean fantasy in terms of fairies and elves and goblins, but in the sense of wholly created “worlds” that feel complete.  One of the first I recall reading was the Earth-Sea trilogy which took place on a planet very unlike our own.

Anyway, the latest series I’ve been sucked into, like many is “The Hunger Games”.  For those of you who live under a rock and have missed all the hoopla, it is set in a future dystopia where “Districts” are required to send a male and female “tribute” to the “Capitol” to participate in gladiatorial combat to the death.  The opening scenes remind me much of Shirley Jackson’s “The Lottery.”

In any case, at a couple of points, the story’s hero, Katniss Everden is told by her drunken mentor, Haymitch Abernathy, “Stay Alive”.  On the face of this, when heading into near certain death, this advice from a drunk seems rather pointless.  After all, isn’t everyone in the “Games” trying to Stay Alive?  She at first dismisses him as a drunken fool.

However, as time goes on, despite not being able to communicate directly with him, she starts to understand him better.

And while never fully stated, I believe she finally realizes his advice wasn’t nearly as obvious as it sounds.  It is rather more like a Zen Koan.  By staying alive, she’ll live.

The first time she applies this lesson, without fully realizing it, is right when the games begin.  Unlike many of the tributes who go for the cache of items the Gameskeepers provide at the start of the game, she grabs one or two items and flees for the woods, barely staying alive in the process.  However, she learns that night that 11 of the 24 people who started the game died in the initial moments, most of them trying to grab items from the cache.  They had failed to stay alive.  More importantly, they had failed to follow the advice of “Stay Alive”.  Rather, while planning for the future (“If I can get these weapons now, I can use them later on”) they failed to take into account the present.  In the present there were 23 other tributes intent on killing them.

Soon Katniss realizes that by focusing on “staying alive” she can actually win the games.  She makes some mistakes, but also does many things right.  Once she’s assured that she can stay alive, only then does the actually go on the offensive.  As a result she at the end, she is still alive.

Ultimately, I realized this similar to the point I often try to drill into people when I say, “fly the plane”.  This reflects lessons learned by the NTSB and others that there are airplane crashes that result as a result of the pilot failing to do the most important job, flying the plane.  They may get distracted, or worse focused on the wrong issue and and end up flying the plane into the ground.

If you don’t believe me, think back to your early days as a driver and how you might have been easily distracted adjusting the radio, or picking up something off the floor.  If you’re like most drivers, you probably had a few near misses where the distraction from driving almost caused an accident, or worse, did cause an accident.

In my most memorable incident, I was in a vehicle with my father, approaching a merge and was trying to downshift.  I was still intent on learning to drive stick and this truck was a bit tricky at times.  But I was determined to properly downshift.  So determined in fact that I ignored the big red hexagonal sign with the bright white letters instructing me as to what I really should have been doing.  I also ignored (well honestly I think I may have snapped at least one reply) my father’s increasing admonishment to do as that sign instructed lest something bad happened.

I can’t recall if I successfully downshifted or not, but I do know that once I returned to the actual task at hand, DRIVING, I was about 40′ beyond said sign and was lucky I hadn’t been hit by a car from the other leg of the merge.  I had been distracted by something that I thought was very important, “downshifting and not stalling out” and missed the real goal at the time, STOPPING.  Or in other words, staying alive.  Sure, downshifting and not stalling out was an admirable goal.  And had the “stopping” part been successfully managed, the proper goal to focus on.  But the “stopping” part really trumped all else.

 

So: Stay Alive first and then focus on winning

 

 

 

My First Science Experiment

Posted on by
Reply

I was thinking the other day about my first science “experiment”.

I was probably 4 years old at the time.  And I wanted to know if the TV basically did, what I now know would be referred to as “caching”.  Specifically, I wanted to know if I turned off the TV and turned it on fast enough if somehow the TV would “remember” the second or two of the episode that was aired while it was off and somehow play it back when the TV was turned back on.

I still remember the process.  It was obvious to me that the TV wasn’t going to remember very much (simple experience showed me that since I obviously couldn’t watch a show that was on an hour previous).  But, I figured if it was quick enough, perhaps somehow it was stored in the TV.

The problem though, was “how to tell?”  So I had to set up some conditions.  Basically it came down to watching enough TV to be able to guess what the next word would be in the sentence the actor or presenter was saying.

So once I determined how to do the experiment, I proceeded to sit in front of the TV and wait for a line of dialog where I figured I could safely guess the next word or two.  Then I’d flip the TV off and on.  I also tried changing the channel to see if that would affect things.

After a few tries, I was pretty much convinced that the TV wasn’t capable of caching anything.  I never could be sure though since I realized my guesses might be wrong.  But, my confidence was high enough that I concluded that when the TV was off, anything transmitted to me was lost.

So, at age 4 or so, I had somehow already figured out the scientific method and was engaging in science experiments.

That though has pretty much defined my life.  I have to remind folks, I received my BS from the School of Science at RPI, not the School of Engineering.

Scientist: It’s the way I roll.

 

White Ford Taurus

Posted on by
4

So, listening to the 24 hours of SQL Pass webinars. The current topic is “I Was Young and Didn’t Know Any Better” and the panelists are sharing war stories of mistakes they’ve made.

So far they all sound familiar.  So I thought I’d share one of mine.  Well technically not my mistake, but one that I adopted.

Many moons ago, I was advising a company that was involved in building websites for car dealerships.  One day they needed to do an update to the live data.  This was back in the days when all code and updates were cowboy updates.  Of course you ran the query on the live database the first time. You didn’t necessarily have a stating database or even as was later discovered, good backups.

Apparently a customer needed to update a car in their inventory.

UPDATE AUTO set cartype=’White Ford Taurus’

Nice, syntactically valid… and a disaster.  Ayup.  Suddenly every car in the database at every dealership was now a White Ford Taurus.

Ever since then we called that the “White Ford Taurus” problem.

Now, I might mock doing updates on live data, but sometimes its necessary.  I’m curious how others prevent their own “White Ford Taurus” problems.

Personally, I just now make EXTRA effort to make sure I have a WHERE clause.

But I also tend to almost always do it as:

BEGIN TRAN
UPDATE AUTO set cartype=’White Ford Taurus’
if @@rowcount<> 1 rollback tran else commit tran

Or sometimes I’ll simply hardcode the rollback tran, run it once, see what happens and then rerun it with a commit tran.

So, if rather than updating the 1 row I want, I find myself updating 1000s of rows, I’ll catch myself and be safe.

Sure, it’s not perfect, both it and using the WHERE clause require me to make sure I don’t forget them.  But the more ways to catch it, the better.

Obviously avoiding ad-hoc updates on live data is preferable, but when you can’t, be extra careful.  And of course make sure you have good backups. But that goes without saying.

 

 

Processes

Posted on by
Reply

It’s funny.  In my personal life, I’m a pretty casual person.  I don’t generally create grocery lists.  I don’t write detailed lists of things I need to get done.

That said I’m a HUGE fan of “process” when performing tasks that absolutely have to be done in a specific manner.  In my old job, I often had to do complex updates on web and databases servers with zero downtime.  In some cases, this is like replacing the engines on a 747 while it’s in flight. In cases like that I or my staff would create what we called a “CRP – Change Review Plan”. (I wanted to call them Change Review Analysis Plan, but I decided I didn’t want to take CRAP from anyone.)

Anyway,  a book I would highly recommend is The Checklist Manifesto. This delves into this concept far more than I can here.

However, one thing I learned years ago was when too much process actually can make things worse. There’s a story, possibly apocryphal, of an incident during the servicing of the space shuttle many years ago.  In the VAB while it was being rotated from the horizontal position to the vertical for attachment to the ET, a loud klunk was heard from inside the engine compartment.  Now, one doesn’t have to be a rocket scientist to realize that a loud klunk is probably NOT supposed to happen during this procedure.

So, what had happened?  Had a procedure been violated?  Well, in reality yes.  On paper no.  In order to provide quality control, almost anything NASA touched when it was servicing a shuttle would get signed off on by at least one if not multiple people.  Supposedly there was a checklist that at least 5 people signed off on to ensure that all tools had been removed from the engine compartment.  Sure enough, that list had 5 signatures on it.  However, the tool, I believe a wrench, now sitting at the aft end of the shuttle proved otherwise.

More recently (like tonight) I was reminded of this as I sat in on a meeting. The meeting was at a local college student club and the purpose was to discuss the fact that some unauthorized people may have gained access to an area they were not allowed access to.  There was some good discussion of what had occurred and how to avoid it in the future.

At one point someone suggested, “How about a video cam or something so the folks sitting at the desk can check to make sure the room really is empty?”  That’s a nice high-tech solution. But it was honestly in search of a problem.  The real problem appears to be that the people at the desk weren’t doing their job properly in the first place: making sure doors are locked and checking proper IDs.  I pointed out, adding yet another task to their job description was unlikely to solve the root problem and was unlikely to have kept the unauthorized people out.

Ultimately, it looks like the approach the students will take is honestly, probably the simplest one: Asking to change the door lock so that the desk person isn’t responsible for locking it, but rather make it autolocking.  This way, when the last authorized person does leave, it is locked automatically.  No additional processes are required and in fact the existing ones are simplified and made more failsafe.  The door in question should now be locked when it should be, whether or not the desk person checks it as they are supposed to.

Sometimes, the simplest solutions really are the better ones.

 

 

Customer Service: “We aim to please.”

Posted on by
Reply

So, I’m sitting on the train today, when one half of the couple behind me returned from using the lavatory and remarked to her partner, “Don’t use the bathroom on the left.”  Apparently the previous user had been polite enough to put the seat up.  But not polite enough to actually aim.

All I could think was how nice it would be if the train had QuiCR on-board.  Within seconds she, or even myself having overheard the situation could have reported the issue and a ticket created.  That ticket could then either be handled immediately upon arrival at the destination, or perhaps in the meantime an email sent to the conductor so he could have closed the lavatory for the reminder of the trip; thus preventing any other unfortunate patrons from being exposed to those conditions. 

Quick feedback means a QuiCR response and a QuiCR response means a higher level of customer satisfaction. Think about it.