Git ‘r Done

Posted on by
1

It’s rare I post items so quick in succession, but I’m trying to post a bit more often and these topics work together.

I mentioned in my previous post about the group of people I work with on the NCRC Educational Committee.  But I wanted to follow that up with a comment about a goal

In cave rescue, our goal is to get the patient to the surface as quickly and safely as possible in as good or better shape than we found them.

Ultimately that goal should drive pretty much everything we do on a rescue.

Sometimes though, students fail to see it that way. On one hand it is amusing when we watch students take a simple problem and over-complicate it. Sometimes two instructors will look at each other and ask, “why are they doing it THIS way and not THAT way?” During training it’s easy to refocus them and remind them what the goal is. At the end of the week of training we have a mock rescue where the students are on their own. At this point, if they lose focus of the goal, they may take longer than they expect or wish to.

During one practice I was on as a studnet, a discussion began about how to move the litter with the patient in it under a tight low roof along a stream passage. After a minute or two of discussion, 2 of the other students and I looked at each other and realized the other members of the group were too focused on convincing the other members that their way was the best way to move the patient. The entire time they were trying to “win” the conversation, which had apparently become their current goal; the patient wasn’t moving.

So, the three of us simply moved the patient quickly and safely to the other side of the obstruction. After about a minute, the conversation stopped and the folks on the other side of the discussion realized the patient they were arguing about moving, had been moved. Things improved from there.

Now, by no means should it sound like I’ve never lost focus (see my post The Hunger Games for an example of a potentially more dangerous situation where I definitely lost focus of the correct goal.)

But this leads to a question: “How should the goal be accomplished?”

To give an example, perhaps I can build and operate a beautiful 4:1 haul system where every leg collapses the optimal amount and I can operate it with just 2 people.  Or, I can put up a 1:1 haul system that’s inelegant and requires 6-7 people to operate it.  Both will move the patient, but which one is “better?”

Well, honestly, “it depends”.  If I have plenty of extra people and I’m short on time, I’ll go with the 1:1 almost every time.  It’s simple and it works.  It can be setup in just a few minutes and requires very little equipment.

But what if I’m tight on people and I have the time?  Perhaps then the 4:1 is the proper solution.

This is where experience and judgement come into play.  Both systems “Git ‘r Done” and both can help me with my goal of getting the patient to daylight. And that is my goal in a rescue.  My goal in a rescue is NOT to build a beautiful 4:1.  My goal is to build a system that gets the patient out safely and quickly.  If a 4:1 will work best to accomplish the goal, I’ll do that.  If it won’t. I’ll forgo it, no matter how sweet and sexy it may seem to me.

We sometimes teach students a handy metric of two questions to ask themselves:

  1. Does it work?
  2. Is it safe?

A 4:1 that isn’t fully rigged when the patient arrives fails the first question, no matter how elegant and well it may operate when its finally rigged.

On the other hand, if the 1:1 is fully rigged, but I don’t have enough folks to operate it, it also fails the first question.  However, if I have enough folks to operate it, I’m not going to start discussing how there might be a better way to rig it with fewer people.  Note that “is it the BEST or OPTIMAL solution” isn’t part of the metric.  In this case, it really doesn’t matter.

Keeping these questions in mind can often negate extra conversation (such as the example above of the patient not moving while folks were discussing the BEST way to move him.)

So, when you face the task of solving a problem, especially one with time pressure and that is most likely a one-off, ask yourself if the solution you currently have is safe and if it works.  If you can answer yes to both, then Git ‘r done.

Smart People

Posted on by
Reply

Like many, from time to time, I’ve had the distinction of being “the smartest person in the room”. (Often that’s when I’m the only guy in the room, but that’s just a minor detail.)

This past weekend though, I had the pleasure of being in a room full of people where I was definitely not the smartest guy in the room.  One of my side activities is working with the Educational Committee of the National Cave Rescue Commission. This entails, among other things, having face to face meetings once or twice a year. During this time we work on the curriculum, trying to improve it every year.  This weekend’s meeting had 8 people (including myself) in attendance.  The people there bring an extremely wide degree of skills to the table, ranging from medical experience, SRT experience, grammar experience, experience about ropes and devices, rescue experience and more.

With such a diverse background, there are times when not everyone is in agreement on various teaching points.  But, while there may be spirited debate at times, everyone still keeps the end goal in mind: developing the best possible curriculum for cave rescue.

However, one has to be careful about “the smartest guy in the room.”  There is an old joke about a plane about to crash. It has 3 passengers, a Boy Scout, a priest, and a Nobel Prize winner and 2 parachutes.

The Nobel Prize winner grabs a pack shouts, “My discoveries will save the world, I deserve to live” and jumps out of the plane.

The priest tells the Boy Scout, “Son, please take the last parachute.  I have lived a good life and I am prepared to meet the Lord”.

The Boy Scout turns to the priest, “Don’t worry Father, the smartest guy in the world just grabbed my backpack and jumped out of the plane.”

Being smart doesn’t make one immune from error.  But surrounding oneself with smart people can often lead to better solutions.

You don’t have to be the smartest person in the room, but you should at least try to surround yourself with them.

Think your way

Posted on by
Reply

Years ago Thomas J. Watson at IBM thought up a single word slogan: Think!.

Later Apple countered with Think Different!

I’ve always been fascinated with how people think.  One thing I’ve learned over the years is most people don’t think nearly as much as they think they do.  This isn’t necessarily a bad thing.  Actually in many ways it’s most likely an evolutionary adaptation that for the most part works.

For example, if a million years ago, you suddenly saw your ancestors all running away, you’d probably leap up and follow them without waiting to confirm there was a sabre-toothed tiger hot on their heels.  By not taking the extra second or two to think, you were able to pass on your genes.

But, in today’s highly evolved society, often that exact same reaction is not the right move.

I was reminded of this today getting off the train in New York City.  I tend to sit in the middle of the car (a lot of thinking has gone into this actually!)  So when I get up, I always have a choice of which way to go to get off the train.  Now, in a perfect world, it wouldn’t matter.  Since I’m equidistant from both ends of the car, it shouldn’t matter which way I go.  But often it does.  Today actually I was a little forward of middle.  I turned back to see more than 1/2 the car lined up behind me heading towards the rear of the car.  In part this was because the stairs from the platform towards the station are in that direction.  

Everyone was standing still.  I walked forward and was out of the car and to the stairs before the rear of the car had barely emptied.  I’m not sure what the hold up was. But the fact was, more than 1/2 the car apparently saw someone else get up, head towards the back so they did.  They never took the split second to look forward to see if that would be any faster.

Now you may be saying big deal. And you’re right. Getting off a train car a few seconds ahead of other people usually isn’t.

But, I’ve heard of similar things happening in more critical situations. How many times have you listened to the flight attendant or movie announcer point out fire exits and then promptly forgot about them?  How many folks make notes of all the exits in a building when they enter?

A study of fires and incidents in public facilities often shows that folks tend to do one of two things. 

One, they go where everyone else is going.  Again, this makes sense at a basic level.  There is the underlying assumption whether they realize it or not that they’re following the person in front of them because they assume the person who is moving must know something they don’t.  Of course it never dawns on them that the person in front of them may simply be moving in that direction because of the person in front of THEM.  The tragic side effect is that often you get crowding, trampling or simply wedges by the exit.  Often you’ll find a number of deaths occur near the most visible or most used exit to a facility.  In many cases this is due to wedges of people getting stuck trying to get through the door.

The other thing that people tend to do is head towards the exit they know about.  In fact this is most likely what causes the person at the front of the pack above to start moving in the direction they’re moving. 

Unfortunately, building codes generally assume that in an emergency, folks will exit all available exists (or some subset) in roughly equal numbers.  Experience shows in a crisis, this is rarely the case.  (I do believe this assumption is also starting to change and some building codes are changing to reflect that.)

(as an aside, one of the more remarkable survival stories I recall from 9/11 was a small group of people that essentially MADE their own exit by cutting a hole through a fireproof wall to another area where they could escape from.  So, sometimes consider making your own exit.)

Now, since I hate to end these passages on a somber note and because I want to relate this more to the IT industry, I’ll point out that often the successful businesses are not the ones that do what the heard does.  They do it differently. 

So, two exercises left to the reader.  Next time you go to a movie theater or restaurant, take a note of the exits and decide which one you will head to that you might have not noticed before.  Or if you’re stuck on a train or in a long line, stop to think if perhaps there’s another way people are overlooking.

The other one is more fun and one friends did in high school.  At the next group photo, pretend to see something and point to it.  See how many people follow along because obviously if you saw something, they must see something.  Then ask the last person who pointed what they were looking at.  I’ll bet in most cases you’ll hear “whatever it was they were looking at!”

 

A Solution Exists

Posted on by
Reply

I’m reminded of an old, very geeky joke:

A mathematician, a scientist, and an engineer all take a room at a hotel.  They each wake up separately to discover a small fire in their wastebaskets.

The scientist sits for a minute, calculates the amount of heat being created by the fire, determines the amount of water required to put it out, fills up his ice-bucket to that precise amount and tosses it on the fire, putting it out.  There’s no excess water and the mess is minor so he goes back to sleep.

The engineer looks at the fire, grabs the ice-bucket fills it to the top with water, figures there’s more than enough water in it, toss it on the fire with a margin of water to spare.  The fire goes out and he goes back to sleep.

The mathematician looks at the fire, looks at the ice-bucket, determines a solution exists and goes back to sleep.

When solving a problem, sometimes knowing a solution exists (or doesn’t exist) is half the battle.

I’m currently trying to win a programming contract to update a gentleman’s website. One of the requirements may be to integrate with some sort of payment service.  So, some quick research showed such a solution “existed”.  I had achieved the level of mathematician.  The service suggested is  Paypal’s https://www.paypal.com/ipn service. 

But it’s one thing to show a solution exists, I had to actually work with the solution to make sure if the time comes, I can implement it.  So, spent a few hours tonight working with the code example provided by Paypal and a database I created to test the proposed solution. After a couple of minor errors, I was able to use the test sandbox to call my callback URL and record the responses to the database.  I had achieved the basics. But this was also the hardest part.  So I have achieved the level of scientist here.

Now, if the customer gives me the go-ahead, I’ll have to expand the code to actually parse the request string and other details and integrate it with his code and database.  That said, that part is actually fairly trivial in many ways as I will control the entire process there.

So I haven’t quite achieved the level of engineer, but if the time comes, I will.

So that fire is out.  On to the next one.

QuiCR’s latest product

Posted on by
Reply

I mentioned in my latest post I was working on a new project, one that valued simplicity over complexity.  I can now talk about it a bit more.

I had approached a local cab company about the QuiCR product.  Unfortunately, given who his largest market demographic was, (most of his fares do not have cell phones, let alone smart phones) we decided it wasn’t a great fit.

But, as I mentioned, he had made an off-hand comment about something he would like: namely an ability to allow a smaller demographic of his, the local college students, the ability to send a text to his dispatcher and request a cab.

This is one of those design ideas that’s both deceptively simple and complex at the same time.  It’s simple because “Receive text, display text, allow a response” about describes the problem.

Now, the simplest solution obviously would be a to give the dispatcher a cell phone with texting capabilities.  That would also be the wrong answer.  For one thing, his dispatchers work at a frantic pace and time is off the essence.  While some folks may be able to whip off text messages using “text-speak” in seconds, his staff isn’t among those with fingers that nimble.  

It also doesn’t provide for easy reviewing of messages and threads and the like.

So, the trick was coming up with a computer interface that was simple enough that it could be adopted with only a few minutes of training and that wouldn’t interfere with their current manual dispatch system.

The keyword there is manual. Yes, there are systems out there with all sorts of bells and whistles that can integrate with GPS, credit card systems, IVR and much more.  Those systems also costs a LOT of money.  And in at least one case, a vendor was suggesting that to adopt it, he hire another dispatcher to handle the increased load.  Note the load wasn’t necessarily from increased business, but simply from the complexity of the system.  Now, don’t get me wrong, in a large city where you have dozens of cabs, such a system is the right approach and scales well.  But, it doesn’t scale very well to smaller companies.

His dispatchers use a very manual system.  And it works. Hopefully my new “text-dispatcher” will integrate well with the current system and generate some new business for him.

Sometimes, simpler is better, but harder

 

Timers

Posted on by
Reply

There’s a photo I recall of the Space Shuttle flight deck looking out over the cargo bay.

What’s most interesting about it is the number of kitchen timers velcroed to various surfaces.  They were definitely afterthoughts.

After a few flights NASA started to realize that their multi-billion orbiter was short a few $1.50 timers on the flight deck.  They were used for things like timing rendezvous maneuvers, arm movements and more.  It was such a simple  need that was evidently overlooked early on.  Or perhaps not so much overlooked as much as no one foresaw the usefulness.

I was thinking of this tonight as I was cleaning up in the kitchen and realizing how useful timers are so I don’t forget things. In this case, putting the wok on the hot stove to boil off some water.  Something I could easily walk away from and realize an hour later I had forgotten about. Sure a pretty mundane use, but a very important one.

I’m also reminded of it as I’ve submitted a proposal to a local company to add something new to their business.  The idea came from an off-hand comment the business owner had made.  More importantly though, he liked the prototype because I listened to his other comments.  This had to integrate into an existing manual system as easily as possible.  He had others come to his business and suggest multi-thousand dollar solutions that would require he hire MORE staff and send them all to training.  That was not a winning business solution.  If my proposal is recommend, it will involve a laptop on the desk and at most 1-3 mouse clicks to do what he wants his team to do.  Time in training… 5 minutes. Cost.. under $1K upfront (and under $1K monthly).  Possible impact on his business >$1K a month.

That’s hopefully a winning combination.  I’ll know tomorrow if it works.

But sometimes the simplest approach is the best approach.

 

When things aren’t the same

Posted on by
1

A recent thread on the Nanog mailing list regarding the outages this past weekend reminded me of one of my war stories that I thought I’d relate.

About 7 years ago, I was involved in a project to help a different division of the company upgrade and replace their cluster and SAN with new hardware.  I was in a fairly hands-off mode, but towards the end was more involved.  We had outsourced the hardware and software installation to another company, but we were about to take over and move the production system to it.

I tend to be a bit paranoid when it comes to changes.  Changes can introduce new, unknown points of failure.  There’s a reason for the saying that “better the devil you know.”

Anyway, before we were about to go live, I asked, “Has anyone tried a remote reboot of the box, just to make sure it’ll failover correctly and come up?”

“Oh sure, we did that when we were in the datacenter with the vendor a couple of weeks ago.”

Now, there had been several changes to the system since then, including us adding more components of our software.  So I did a quick poll and asked if everyone was comfortable with moving forward without testing it.  The answer was in the positive.

Still, as I said, I’m a bit paranoid.  So I basically asked, “Ok, that’s nice, but do me a favor, reboot the active node anyway, just to see what happens.”

So they did.  And we waited.  And we waited.  And we waited.  As I recall, the failover DID work perfectly.  But the rebooted node never came up.  So we waited.  And waited.  Now, unfortunately due to budgetary reasons, we didn’t have IP-KVM setup.  We had to wait until we could get to the remote datacenter to see what was going on.

Meanwhile, the vendor that did the install was called and they assured us it was nothing they had done. It had to be a hardware issue, since they had installed things perfectly.

That’s your typical vendor response by the way.

Finally we get someone out to the datacenter.  What do they see, but the machine waiting to mount a network share to get some files.  This puzzled us, since we had no network shares for this purpose.  We pondered it a bit.  Then it dawned on me that the only thing different between our reboot and the earlier one, was the vendor was in the datacenter.

So, we made the necessary changes to not try to mount network files, tested the reboot a few times and all was well.

One of my team called the vendor and asked, “hey, when you do an install, does your tech mount stuff off of their laptop?”  “Oh sure, that’s how we get some of the critical files where we want them, why?”

So, one little change, having the vendor there (with their laptop on the network) and not having the vendor their made all the difference.

If you’re going to test, test in real-world scenarios as much as you can.  Make sure nothing extra is plugged in, even if you think it won’t make a difference.

If you can, test it just the way you would expect it to fail in the real world.  If that means you have redundant power supplies, test it by tripping the breaker on one leg.  If you find yourself saying, “Oh no, that’s too risky” guess what, you’re not nearly as protected as you think you are.  If you can suggest doing the test with confidence you’re 1/2 way to success.  If you actually DO the test and things work, you’re all the way.

In conclusion, one seemingly innocuous change can make a huge difference.

 

Change is good

Posted on by
Reply

So, for the “fun” of it, I added some dns names to my home network (namely the cablemodem and the router).  Now, it’s not like I really NEEDED to do so. The honest truth is IPv4 network addresses are rather simple to remember. So, I could just type in the IPv4 address of the device I want to reference and do fine.

So why bother?

Well, a former colleague of mine convinced me that it was time to change.  You see, IPv6 is coming.  I won’t bore anyone with the details, but let’s just say that I’m not about to try to remember the IPv6 addresses of any of my devices.  At first, the Luddite in me rebelled at the idea that I could no longer just refer to devices by IP address.  I attempted a few feeble attempts at arguments (hey, it’s easy to remember the IP address of some public DNS servers for example, and that’s one of the few times in an IPv4 network it really helps to have memorized some IP addresses.)

But the truth is, over time, I saw his point.  IPv4 was the old way.  IPv6, for a variety of reasons is the future.  My desire to be able to say I could remember the IPv4 address of my router was simply stubbornness.  The truth is, there really isn’t any real benefit to it.  The fact that I could (and did) reference my network printers by IP address wasn’t really helpful.  In fact, it meant that when I wanted to move it to a different address, I had to go to individual machines and make changes.  Now, I had  a small network, and only changed the printer address once in the past 5 years.  So it wasn’t a huge burden.

But now that I’ve moved to using hostnames, even for stuff I used to use IP addresses for, my limited memory can be used for more useful things.  And when I do reconfigure my network (say add another printer, or expand it in other ways) a simple DNS change and I’m all done.  I don’t have to go from machine to machine to make changes.

So, the moral of the story is, just because IPv4 made something EASY to do, it didn’t make it the right thing to do. IPv6 forced me to reconsider my thinking and in the end change for the better.

Sometimes, being forced to change can make you a better person.

SQL Server User Group

Posted on by
Reply

Not much of a post tonight.  I didn’t realize it had been over two months since I last posted.  I’m sure all my faithful readers (all 1 or 2 of them?) have been holding their breath.

Anyway, tonight went to the local SQL Server Users Group meeting.  They had a remote demonstration on query tuning, one of my favorite topics.

In fact performance tuning in general has been a topic I’ve often enjoyed.

First rule: There is always a performance bottleneck.  This is one of the first rules I recall reading.  At first people object, “but the system is fast enough.”  That may be true.  BUT, there is still limiting it from being better.  Of course you might not be able to fix that limit.  But more importantly, it may not matter.

Second rule: It may not matter.  If your query is already running in subsecond times, it may not be worth spending any time on optimizing it any further.  Or, if your query takes 1 hour to run, but runs at night when nothing else is running, it may not matter.

Third rule: Optimize only what you need to.  A classic example of this I’ve seen is finding reports that run overnight and slow down other processes.  You start to optimize it and then think to ask, “is anyone still using this report?”  You find out the report is no longer being used.  Now you can achieve the holy grail (and perhaps the only exception to rule 1): infinite optimization.  Delete the report and suddenly you have infinitely optimized it.

BTW, these rules don’t apply just to SQL Server.

Optimize your life and enjoy more of your time doing things you enjoy.

The Hunger Games

Posted on by
3

I’ll admit it, I’m a sucker for “fantasy” worlds. I don’t necessarily mean fantasy in terms of fairies and elves and goblins, but in the sense of wholly created “worlds” that feel complete.  One of the first I recall reading was the Earth-Sea trilogy which took place on a planet very unlike our own.

Anyway, the latest series I’ve been sucked into, like many is “The Hunger Games”.  For those of you who live under a rock and have missed all the hoopla, it is set in a future dystopia where “Districts” are required to send a male and female “tribute” to the “Capitol” to participate in gladiatorial combat to the death.  The opening scenes remind me much of Shirley Jackson’s “The Lottery.”

In any case, at a couple of points, the story’s hero, Katniss Everden is told by her drunken mentor, Haymitch Abernathy, “Stay Alive”.  On the face of this, when heading into near certain death, this advice from a drunk seems rather pointless.  After all, isn’t everyone in the “Games” trying to Stay Alive?  She at first dismisses him as a drunken fool.

However, as time goes on, despite not being able to communicate directly with him, she starts to understand him better.

And while never fully stated, I believe she finally realizes his advice wasn’t nearly as obvious as it sounds.  It is rather more like a Zen Koan.  By staying alive, she’ll live.

The first time she applies this lesson, without fully realizing it, is right when the games begin.  Unlike many of the tributes who go for the cache of items the Gameskeepers provide at the start of the game, she grabs one or two items and flees for the woods, barely staying alive in the process.  However, she learns that night that 11 of the 24 people who started the game died in the initial moments, most of them trying to grab items from the cache.  They had failed to stay alive.  More importantly, they had failed to follow the advice of “Stay Alive”.  Rather, while planning for the future (“If I can get these weapons now, I can use them later on”) they failed to take into account the present.  In the present there were 23 other tributes intent on killing them.

Soon Katniss realizes that by focusing on “staying alive” she can actually win the games.  She makes some mistakes, but also does many things right.  Once she’s assured that she can stay alive, only then does the actually go on the offensive.  As a result she at the end, she is still alive.

Ultimately, I realized this similar to the point I often try to drill into people when I say, “fly the plane”.  This reflects lessons learned by the NTSB and others that there are airplane crashes that result as a result of the pilot failing to do the most important job, flying the plane.  They may get distracted, or worse focused on the wrong issue and and end up flying the plane into the ground.

If you don’t believe me, think back to your early days as a driver and how you might have been easily distracted adjusting the radio, or picking up something off the floor.  If you’re like most drivers, you probably had a few near misses where the distraction from driving almost caused an accident, or worse, did cause an accident.

In my most memorable incident, I was in a vehicle with my father, approaching a merge and was trying to downshift.  I was still intent on learning to drive stick and this truck was a bit tricky at times.  But I was determined to properly downshift.  So determined in fact that I ignored the big red hexagonal sign with the bright white letters instructing me as to what I really should have been doing.  I also ignored (well honestly I think I may have snapped at least one reply) my father’s increasing admonishment to do as that sign instructed lest something bad happened.

I can’t recall if I successfully downshifted or not, but I do know that once I returned to the actual task at hand, DRIVING, I was about 40′ beyond said sign and was lucky I hadn’t been hit by a car from the other leg of the merge.  I had been distracted by something that I thought was very important, “downshifting and not stalling out” and missed the real goal at the time, STOPPING.  Or in other words, staying alive.  Sure, downshifting and not stalling out was an admirable goal.  And had the “stopping” part been successfully managed, the proper goal to focus on.  But the “stopping” part really trumped all else.

 

So: Stay Alive first and then focus on winning