Category Archives: mistakes

… Other Duties as Assigned #2 in a series

Posted on by
Reply

I wrote last year about how my job title seems to be DBA and other Duties as assigned. A little incident yesterday got me thinking about that again. This time though, it got me thinking more about the DBA side of that title.

I’ve mentioned before I’ve written a number of ETLs for various clients. Some are fairly complex and some are fairly simply PowerShell scripts. In most cases, at the end of the script I send an email with a success or failure status and some additional information if appropriate.

Over the last two weeks I had noticed that the email for one particular ETL was coming in later and later. I finally found somet time to investigate. I looked at the destination table and noticed that some of the tables had extremely large numbers of rows. Now, my first inclination was that the source data had increased (which would be a good thing, it meant the client was basically selling more widgets). But the increase seemed too dramatic and large. My next thought was perhaps the export itself was simply giving me more data.

So I decided to look more closely at the data and I noticed something interesting. Picking on of the tables at random, I simply did a 

select * from RandomTable order by GoodIndex

SSMS returned approximately 24,000 rows. But something stood out. There appeared to be duplicates rows. Lots of them. This raised a lot of suspicions.

I then ran the query on my UAT box which in this case actually loads a copy of the production data. I got back about 1,600 rows. VERY curious.

Back on production now I ran

select distinct * from RandomTable order by GoodIndex

This time I got back the same number as UAT. Extremely curious.

Then it dawned on me, about 2 weeks ago, we had made a change to where the script looked for the source data, a number of CSV files. UAT had not changed, but Prod had. The reason for this change was to be able to get the ETL PowerShell script to run on the proper production server (the original location had security issues reading from the original CSV file location.)

So my first thought was that the team that had updated their export had somehow left out the command to delete the old file and was simply appending. I was about to write a pointed email when I stopped myself.

I looked at my code and realized that when I had setup the script on the new production box, I had properly handled all the permissions except one: the step that truncated the destination table. Sure enough, my code was no longer truncating the table before I inserted new information. Fortunately I realized my mistake before sending that pointed email.

So was this a DBA issue or a “other duties as assigned” issue? I don’t know, but I’d say as a DBA I should have not created the problem in the first place, but thankfully, due to proper logging and emails I was able to catch it and solve it fairly quickly.

Moral of the story: Pay attention to details. Notice when things start to drift. They can be the sign of a larger issue.

Stuck, with Responsibility

Posted on by
1

So, by now, you may have all heard about the vehicle that got stuck trying to go through a somewhat narrow passage. No, I’m not talking about the container ship known as Ever Green. Rather I’m talking my car and the entrance to my garage!

Yes, due to circumstances I’ll elucidate, for a few minutes the driver’s side of my car and the left side of my garage door opening attempted to occupy the same spot in space and time. It did not end well. The one consolation is that this mishap was not visible from space!

Now I could argue, “but it wasn’t my fault! My daughter was driving.” But that’s not really accurate or fair. Yes, she was driving, but it was my fault. She’s still on her learner’s permit. This requires among other things, a licensed driver (that would be me) in the vehicle and observing what she was doing. She did great on the 8 mile drive home from high school. So great in fact that when she paused and asked about pulling into my garage, I said “go for it.”

To understand her hesitation, I have to explain that the garage is perpendicular to the driveway and a fairly tight turn. It’s certainly NOT a straight shot to get in. I’ve done it hundreds of times in the last 5 years (when the garage was added to the house) and so I’ve got it down. Generally my biggest concern is the passenger side front bumper “sweeping” into the garage door opening or the wall as I enter. I don’t actually give much thought on the driver’s side.

So, I gave her the guidance I thought necessary: “Ok, stay to the far right on the driveway, this gives you more room to turn.” “Ok good, start turning. Great. Ok. Ayup, you’ve cleared the door there, start to straighten out.” “Ok you’re doing…” Here the rest of the cockpit voice recorder transcript will be redacted other than for the two sounds, a “thunk” and then a “crunch”. The rest of the transcript is decidedly not family friendly.

The investigator, upon reviewing the scene and endlessly replaying the sounds in his head, came to the following conclusions:

  • The “thunk” was the sound of the fold-way mirror impacting the door frame and doing as was intended, folding away.
  • The “crunch” was the sound of the doors (yes, both driver’s side doors) impacting the said door frame.
  • Both the driver and the adult in charge were more focused on the front passenger bumper than they were on distance between the driver’s side and the door frame. Remedial training needs to be done here.

Anyway, I write all this because, despite what I said earlier, in a way this is a bit about the Ever Green and other incidents. Yes, my daughter was driving, but ultimately, it was my responsibility for the safe movement of the vehicle. Now, if she had had her license, then I might feel differently. But the fact is, I failed. So, as bad as she felt, I felt worse.

In the case of the Ever Green, it’s a bit more complex: the captain of a ship is ultimately responsible for the safe operation of their vessel. But also, in areas such as the Suez Canal, ships take on pilots who are in theory more familiar with the currents and winds and other factors that are local to that specific area that the captain may not be. I suspect there will be a bit of finger pointing. Ultimately though, someone was in charge and had ultimate responsibility. That said, their situation was different and I’m not about to claim it was simply oversight like mine. My car wasn’t being blown about by the wind, subject to currents or what’s known as the bank effect.

What’s the take take-away? At the end of day, in my opinion and experience, the best leaders are the ones that give the credit and take the blame. As a former manager, that was always my policy. There were times when things went great and I made sure my team got the credit. And when things went sideways, is when I stood up and took the blame. When a datacenter move at a previous job went sideways, I stepped up and took the blame. I was the guy in charge. And honestly, I think doing that helped me get my next job. I recall in the interview when the interviewer asked me about the previous job and I explained what happened and my responsibility for it. I think my forthrightness impressed him and helped lead to the hiring decision. The funny part is, when I was let go from the previous job, my boss also took responsibility for his failures in the operation. It’s one reason I still maintained a lot of respect for him.

So yes, my car doors have dents in them that can be repaired. The trim on my garage door needs some work. And next time BOTH my daughter and I will be more careful. But at the end of the day, no one was injured or killed and this mistake wasn’t visible from space.

Stuff happens. Take responsibility and move on.

Guy’s it’s on Us

Posted on by
Reply

A short thread on Twitter yesterday prompted today’s blog. Dr. Jen Gunter (who I do not follow) mentioned her planned response to “I don’t have a question, it’s more of a comment.“

One of the replies I thought completely missed the point and I tried to respond in a somewhat humorous but pointed way to the man replying. It took him about 9 hours, but he finally replied and I think based on his reply, completely missed my point. Oh well. I had tried.

But it got me thinking. Had I been blunt enough? Should I have been a bit more confrontational? Could *I* stand up at a conference and give Dr. Gunter’s pointed reply? And of course, the nagging question in the back of my head, “had I commented when I a question was the appropriate response?” or otherwise ended up “centering the discussion around me rather than the original person?”

Let me address the last first. I’m sure I have. I like to think “well I’m a friendly guy, I like to relate and show I’m relatable.” And that’s all true, but, that’s also part of the problem. It’s a case of recentering a discussion or something around me. I’ve always tried to be conscious of this since college when I took a class at the women’s college down the hill and realized that what I had heard about men dominating discussions was true. In a class of about 20 students, with just 3 men, one of the men (and no, it wasn’t me) clearly dominated the discussion.

Anyway, back to my response. I actually sometimes am jealous of some of my friends who can have “I have no fucks to give” attitude and will openly confront someone like that. I think sometimes that can be a good thing, especially with a more egregious example. And I’ve seen some that are pretty bad.

My attempt to deflect the one tweeter’s reply with a bit of humor apparently failed. So I started to think about how I might handle this at an actual seminar and then I realized I had.

It was at a SQL Saturday a number of years ago. It was a good topic, though, for reasons unknown to me, the presenter has not, to my knowledge presented again. And then there was the raised hand. It was someone I knew. And, he had more of a question than a comment. Then again about 10 minutes later. And I think probably a 3rd time. Now, he was in now way being mean or malicious. Heck, I think no matter how hard you looked, you’d never find a mean bone in him. He’s genuinely a decent guy.

But, and this is what I think we all need to do, after the talk, I pulled him aside and pointed out what he had done. He as embarrassed and apologetic. And he vowed to do better.

And as I write that, I realize, this happened TO me. See, I said I wasn’t perfect and I had failed. This time it was on Twitter. I typed a reply that I meant to be supportive and add a touch of humor. A friend DM’d me, “Really?” At first I was confused, but when I asked for her to expand, she pointed out what I had done. Yes, I had meant well, but sometimes intentions are less important than results or even perceptions. I decided to delete my comment, despite her saying it wasn’t necessary. I realized I had not contributed to the discussion and my comment could be a distraction that wasn’t needed. And since them I’ve tried to be better. But a comment she said stood out to me. She DMd me because she thought I was one of the “good ones” that I’d listen and accept feedback. That meant a lot to me. She could have ignored my comment and let me continue to be a jerk at times, or she could have publicly called me out and humiliated me, which might made her point publicly, but caused me to be hurt and not grow. She took the time. I appreciate that.

However, yesterday’s Twitter thread reminded me that all too often in situations like this, women and other minority (in that environment) group end up doing the emotional labor of trying to keep the discussion from recentering the discussion the “I have more of a comment than reply” crow.

Therefore, I think often the onus needs to be on us men to call out our fellow men to say, “hey, that’s not cool” or “do you realize how you came across there? I know you didn’t mean that.” We can’t rely on women and other minority groups to do all the emotional labor. So if you see someone trying to talk over a speaker, pull them aside. If you hear them make an off-color comment in a meeting, speak up. Call out behavior. Find a method that works for you.

I prefer, but am not always good about doing it, calling our behavior a bit more publicly. Not necessarily to embarrass the commenter, but to hopefully get them to correct their behavior and so that the original speaker knows they have support.

For example, if someone in a meeting makes a comment about “yeah, let the girls over in accounting handle it”, unless this is an accounting class for teenagers at an all-girls school, you can and should say “Umm, you mean the women right?”

You don’t have to humiliate a person to make the point. In most cases, the person doing it may not be aware and simply needs a nudge. Give them that chance like I was given. Now, in the end, there will be a few folks that do need to be simply called out and made an example of. I’m ok with that, but for the vast majority let’s work to give them the nudge.

Yesterday was “A Monday”

Posted on by
2

Yesterday was a Monday. I don’t just mean it was Monday, but it was in the Garfield comic sense of things A Monday.

As a consultant, I’ve come to expect certain patterns in my work load. For one client, I know approximately every 2 months, over 2 weekends I’m going to have to patch their SQL Servers. I know certain passwords will need to be updated quarterly or annually. And I know sometimes I’ll have A Monday.

Yesterday was one of those. I woke up, checked my email and noticed two jobs had not run. So I logged in and it appeared that the PowerShell script on each server had hung. I killed it and tried to rerun it, but got an error. This wasn’t entirely surprising. This script, in its first part downloads a file from a 3rd party vendor and last week for example, their SFTP server had been down. At first I expected this to be the problem again. But further testing showed I was getting inconsistent errors. Finally the script ran. But, what normally took about 20 minutes to download, took about 2 hours. We learned later the vendor had done an upgrade to their product over the weekend. This shouldn’t have impacted their SFTP server performance, but here we were. Today (Tuesday) the process took 20 minutes again and is back to normal. Chalk yesterday’s issue up to being A Monday.

Then I took a look at another job that had failed. This one is purely internal. Basically SFTP a file from a Linux server to a NAS for a backup. A quick check showed that the NAS share was inaccessible. Reporting this triggered an avalanche of emails back and forth. The most interesting line basically came down to “Yes, the internal IT team did a migration of the NAS, but the migration was supposed to be completely transparent to the users.” Famous last words in my book. Actually, honestly, what I decided was more disturbing was that the failure was on the new NAS device apparently due to a typo. To me, this means, most likely, all the old shares were recreated on the new device by hand, rather than using a script that read out the old shares and recreated them. In any event, the problem was solved, the job was rerun and the backup created on the now new NAS. Chalk that one up to being A Monday.

Then one of the developers for one of the platforms at this client emailed me and said, “Hey database FOO is in recovery mode, what happened?” This one, fortunately I knew exactly what the problem was. Unfortunately I knew it was my fault. We had decided to reconfigure that database to be a log-shipped copy of the main database and I had set it up over the weekend. I had simply forgotten to set it up to place itself in Stand-by/Read-only mode after it had applied the most recent logs. I’ll chalk that one up to it being A Monday.

All of the above was taken care of before 10:00 AM. The rest of the day was filled with a variety of other issues and items, including looking at a Hyper-V host machine with 16 physical CPUs with hyperthreading turned on hosting 4 VMs, 1 with 4 vCPUs allocated, and the other 3 with 8 each. They’re having performance issues. I’m still tackling that one. Looking at that happened on Monday, but it’s not A Monday issue, it’s been an ongoing issue for months.

So what was it about this particular Monday, or Mondays in general?

Well in this case, all 3 of my early AM issues had one thing in common: upgrades or changes made over the weekend. I’m not going to debate the value or wisdom of the timing here, but just note, that on the particular Monday, it wasn’t just one issue, but three. It was definitely A Monday. But I survived as did my customer.

Now back to my regularly scheduled workload.

 

A Good Guy

Posted on by
Reply

I wrote previously about the dangers of calling yourself an ally. Two completely unrelated incidents in the last week reminded me of that post. Both on their own are rather small items, but I think worth considering.

The first basically happened to a friend at a recent rally in NYC to support the Jewish community. Apparently a young non-Jewish woman accosted an elderly Jewish immigrant at the march for comments he had made about the goal or purpose of the rally. Or to put it another way, a non-Jewish person was telling a Jewish person that the way he was expressing his support for Judaism was wrong. Let that sink in for a minute. Now, to be fair, as a my Jewish friend commented, the young woman’s comments weren’t necessarily technically wrong, but they were out of place.

In the second incident, I replied to a comment a friend had made on Twitter. In reaction she sent me a pair of emojis that equated to, “seriously?” I was confused at first because my tweet had been intended to agree with and support her observation. However, because, as she put it, “I was one of the good guys” she wanted to explain how my reply could be perceived as a form of mansplaining. She realized I hadn’t intentionally tried to overshadow her comments or to be rude. She would have had no problem calling me out in public had that been the case. Instead, she took the time to privately explain to me why what I had done was problematic. I ended up, despite her saying it was unnecessary, removing my tweet because I was no longer comfortable with it. I realized were better ways of I could have replied.

The point of my two examples isn’t to say that the young woman was a bad person, or to self-flagellate myself. The point is that even as a ally, one will make mistakes. This is in part because by not being an actual part of the group in question, one can’t fully internalize what it means to be part of that group and how comments and actions will impact members of that group. But, one can ideally still listen and learn. I appreciate that my friend took the time to explain to me why my tweet was problematic. She was under no obligation to do so. But I appreciate it.

That said, two other quick items: I want to toss a shout out to the South Florida BI SQL Saturday. One can’t go 100% based on names as to how one identifies, but the organizers have tweeted about how they managed to have a 50/50 balance of men and women presenting. It is definitely possible to do this folks.

Finally, a shoutout for my latest Redgate article on Comments and More in PowerShell. This was a fun one to write. I hope you enjoy it.

 

Two Minds are Better Than One

Posted on by
1

I’m going to do something often not seen in social media. I’m going to talk about a mistake that I made. It’s all to common on various certain media sites to talk about how perfect our lives are and how great things are. You rarely hear about mistakes. I decided, in the theme of this blog of talking about how we approach and solve (or don’t solve) problems, I’d be up front and admit a mistake.

This all started with a leak in the downstairs shower. It had been growing over the years and I frankly had been ignoring it.  Why put off to tomorrow what you can put off to next month or even year? But finally, in December of last year it became obvious that it was time to fix the leak. It had continued to grow, and now that my son was home from college for extended break, he had setup a work area in the basement, below the bathroom.  I figured he didn’t really need to suffer from water dripping onto his desk.

So, he and I went into full demolition mode and ripped out the old tile and backer board to get to the plumbing.

New plumbing in bathroom

New plumbing

You can see some of the work here. I also took the opportunity to run wiring to finally put in a bathroom fan. That’s a whole other story.

Anyway, the demolition and plumbing went well. Then we put up the backer board and sealed it. And left it like that. It didn’t look good, but it was waterproof and usable. It was “good enough”. So for about 8 months it sat like that. But with an upcoming pool party, I decided it was time to finally finish it off. One of the hold ups had been deciding on tile. Fortunately, on a shopping trip about a month earlier, my son, my wife and I found tile we liked (my daughter, who ironically still lives at home and will be using the shower more in the next few years than her brother, was in LA on vacation, so she ended up not really having much say in the matter).

So, earlier this month, while the rest of my family took a weekend to go to Six Flags New Jersey, I figured I’d surprise them with finally tiling the shower.  I went to the big box store whose favorite color is orange and bought the required materials. Since tile we had selected is approximately 6″ wide and 24″ long, I had to make sure I got the right mastic.  This was a bit different from stuff I’ve worked with in the past with a bit more synthetic materials in it and it mixed differently, and had slightly different drying characteristics. That, combined with growing darkness lead me to move quickly. The darkness was a factor since any tile cutting I was doing was outside.

And, all that lead to a simple mistake.  On the end wall, there’s a window and as a result part of that wall needed tiles just less than 24″ long. On one hand, this is a huge plus since it means less seams and less places to grout. On the other, it meant in one spot having to work around the trim of the window. And that’s where I made my mistake.  The trim had been put over the original tile, so there was in theory room behind the stool of the window to fit in a piece of tile. That had been my plan.

But, when it came to sliding in the nearly 24″ long piece of tile, it wouldn’t fit. It wouldn’t bend (obviously) to let me get it tucked behind the stool and due to the stickiness of the mastic, I couldn’t slid it in from the top.

So, I cut out a notch. In the back of my mind I somehow was thinking, that it wouldn’t look that bad and tile would cover it.  Well, I was obviously wrong.

In hindsight, I realized I should have cut the tile in two pieces, created an extra seam (like the row below it that had to cover more than 24″ wide in any event) and then I could have slide in the smaller piece and then put in the remaining piece. It would have been perfect, looked great and more likely to be waterproof.

So, this gets me to the title. Had I been doing the work with someone else, I’m sure I’d have said, “Damn, this is gonna suck, any ideas?”

A mistake

My mistake

And I’m sure someone else would have suggested, “Hey, cut the piece and slide it in.”

I like working alone, but sometimes, you need a second person to help. Or more than one brain as I’ve mentioned in the past.  If nothing else, sometimes a duck can help.

Idera Ducks!

A bunch of rubber ducks, including two from Idera!

So, the moral of the story is sometimes two heads are better than one. Oh well, I won’t be the one using that shower, so I won’t see my mistake all the time!

Oh and check out my latest Red-Gate Article on Secure Strings in PowerShell.

“Do What You Love…

Posted on by
Reply

And you’ll never have to work a day in your life.” – Confucius, Mark Antony, Mark Twain.

Honestly, I think that’s some of the worst advice ever. It’s a sure way to end up hating something you love doing. Or if you do follow it, make sure you understand what it is that you love.

I first realized this in high school. I had signed up to do JV soccer, something I enjoyed, but I can’t say I loved. Before school started, we had a day of orientation. It included a hike or run up the mountain behind the school. I loved the woods and I loved (and still do at times) running through the woods. Somewhere along the way, a fellow student saw me running and suggested if I enjoyed running through nature so much, that I consider doing Cross-Country instead as my fall sport. I took their advice; snd hated it for two fall semesters in a row.

I realized what had happened was that I had replaced what to me was a fun, non-competitive activity and turned it into something where I had to perform at a specific level every single time. What I loved was running through the woods, not running competitively.

People assume I love working with computers.  That’s not entirely true. I ENJOY working with computers. I enjoy solving problems and computers are one way I can express that joy. When I make a query run 10x faster, or automate a process that previously took someone an hour a day to do, I enjoy that. But do I love it? Probably not. And I’m actually grateful for that. Because if I loved it, it would mean those days of drudgery where I bang my head against the wall all day trying to solve a problem, or I’m up until 3:00 AM recovering a failed server would turn something I love into something I dread. Something I loved would become a chore.

I love to teach caving. I get a real thrill out of it. But, I suspect that if I spend 40 hours a week, 50 weeks a year doing it, it would soon become a chore.

As my kids started their college journey, I’ve advised them, “find something you enjoy, not something you love. Keep the something you love for your own personal time so it doesn’t become a chore.” And that’s my advice to anyone.

But hey, I could be wrong. What are your thoughts? Did you pick your job because you love it and if so, do you still love it, or did you end up resenting it at all? Or do you enjoy your job?

Tighter than I Remember

Posted on by
Reply

I’ve been reading a book off and on for about a year now called, Being Wrong : Adventures in the Margin of Error. It ties into my interest in meta-cognition and how we think.

We build a model in our head of how the world is. Often times it’s accurate, but often times it’s wrong. In my model of the world, the Sun rises in the East. This model is accurate and corresponds with the model most of us have. But I can easily construct related models that are wrong. Looking back I can remember that it was sunny on a particular day, only to find if I check the records, it was raining. My memory is wrong, but it created the model I had for that day. I can also create a model going forward. Last night, going to bed, the world I was going to wake up to was sunny. This was based on the weather forecast and the fact that the Sun rises in the East. Well, the Sun may have risen today, but it’s not sunny. There’s far too many clouds. So that model was wrong.

This all comes to mind because of a caving trip I did this weekend. There’s a fine little cave near here called Ella Armstrong. It’s really not much of a cave, perhaps 250′ of walkable passage total. But I’ve always liked it even though I hadn’t been there in 15-20 years. It’s got a nice vertical drop at the entrance that’s great for beginners. And it has an easy walk-in entrance.  As a result, I had been considering using it for some cave rescue training next year.

So you can imagine my surprise when two of my fellow cavers who had been there just a few weeks ago said I was in fact wrong and it’s not a walk-in entrance and that in fact it’s rather tight. I was surprised. I didn’t really doubt them, but figured perhaps we had a different definition of what we considered a tight entrance.

Now, let me take a little detour here and mention that almost every caver will at some point joke about how a particular cave has gotten smaller over the years. It’s a little lie we tell ourselves to account for the fact that many of us have put on a few pounds since we first started caving and some passages are in fact harder to get through.  The truth is, rarely due caves get smaller, though sometimes passages sometimes do get larger. There’s a cave in Vermont where I’m pretty sure fitting through 2 of the 3 tightest spots is probably close to impossible for me now. And I know the cave didn’t change sizes, so I’ll have to admit I have.

But back to Ella Armstrong. If you had asked me to describe the entrance prior to this weekend I’d have described it as about 12′ tall at its tallest and 3′-4′ wide most of the way to the top of the drop.

Well, as Saturday proved, the model in my head was only half right. The map lists the entrance as 8′. I’ll call that close enough to 12′ since it simply means there’s no chance to hit my head on the ceiling.

On the other hand, most of the entrance from the surface to the top of the drop is NOT 3′-4′ wide. It probably averages 18″ at most and in parts is just wide enough for me to work my chest through with some contortions. My first reaction when I got to the entrance was, “perhaps some rocks have fallen in and narrowed this from what I remember.” But, after I started to scramble beyond some of the rocks on the surface, I got to the bedrock, which I know hadn’t moved and realized that my model was dramatically wrong. It was tight enough that at one point I considered calling off our trip. It wasn’t that I didn’t think I couldn’t make it further down and past a particular spot; gravity would see to that. It was more that I was afraid I might not get back UP past that spot. Finally after some mental gymnastics I figured a way I could not only get down past the choke point, but felt comfortable thinking I could get back up past it. As I’m writing this from the comfort and safety of my home, you can see my mental gymnastics worked.  Actually, it turned out the crux move on the way out was a bit further up that wasn’t nearly as tight, but had a short shelf about 3′ up in a spot that wasn’t great on foot or handholds. On the way in, that spot didn’t appear to be an issue at all. So again, in the narrow space of about 20 minutes my mental model failed me.

The truth is, we make mental models all the time. It’s how we operate in the world. But sometimes those models are wrong. Sometimes in a positive way, the tight spot getting out wasn’t as bad as I thought, or in a negative way, the spot I thought would be easy, was in fact the hardest sport. But regardless of the errors in many of our models, we generally navigate the world in a successful manner. Being wrong isn’t necessarily the end of the world.

Unfortunately though, as much as I love this cave, it really is too tight to be practical for cave rescue exercises. Which is a shame, because it is a great little cave. And if you’re ever in the area and want to check it out, let me know. I’d love to take you. But I’ll warn you, it’s a bit tight in spots!

JOBS THAT BEAT THE CARING OUT OF YOU

Posted on by
1

Let me start by saying this is NOT an April Fool’s Joke. This is a true story.

I do lay the ‘blame’ for this post squarely two members of my #SQLFamily: first on the heels of Grant Fritchey and his post by he same name. He in turn lays blame on Jen McCown’s post by the same name.

I mention elsewhere in my blog I prefer to be intelligiently lazy, so rather than retype, I’ll post the content from a Quora answer I wrote.  Technically I was just a consultant, and after twice getting a late check I made it clear to them that if they stopped paying me on time, I would stop working.  Apparently they liked me enough that a quick call to the CFO would get me a check cut that day.

So with that:

Let me give you an example of a client I once had. When I started with them, people loved working there and they were expanding and successful. So successful the company got bought.

Then… things changed.

Sales people were finding their expense checks weren’t getting paid (more on that later). Did you know, even if you try to explain to the credit card company that it’s a “company card” if it’s in your name and the company doesn’t pay it, you’ll ruin your credit score? Yes, it’s pretty difficult to be a sales person who can’t travel because no one will give you a credit card any more!

Then, to cut costs, an office move was proposed. Quite frankly, had I not been involved as their IT guy, it would have been a disaster for a variety of reasons. Fortunately for them, besides my IT skills, I could read blueprints. It was quite obvious to me that 2 outlets would not serve an office of 20–25 people with computers and printers. It took me nearly kidnapping the CFO on a day he visited and dragging him to the office to make clear how much more work the office needed. They simply assumed, “oh, it’ll have enough power.”

Meanwhile the previous owner had started a new company (in a completely different industry) and was growing and expanding at a furious rate. Also, my wife was a recruiter at another local company (in a different industry also). The only thing all three of these companies had in common was they all were software related, but the fields they served were completely different.

At one point, the top sales person from the failing company left to go get a job a with the new company. Within days the former company sent a cease and desist letter to the new company insisting they stop poaching employees and if they continued, they’d sue the owner for violating the non-compete clause. Now, keep in mind the owner was very much NOT approaching employees of the old company, but even if he were, the non-compete only applied if he had founded a new company in the same industry. he hadn’t. We had a good laugh at the old company.

Now, meanwhile, my wife, while not exactly poaching, knew that almost any offer she made would be accepted since morale was so bad at the old company.

Then… this happened. I was there for the meeting and sat in on it. It’s the closest I’ve come to “beatings will continue until morale improves” ever.

The CFO and CEO came into town for an all-hands meeting. Their goal was to address, among other things, the late employee expense checks issue.

I will say, they had some pretty looking slides. The slides showed things like cash-flow, moving towards profitability and some other items. But the message was quite clear, “We will continue to pay YOUR expense checks as late as possible because it helps our cash flow. And you should be grateful for this.” They very much could NOT understand why employees were furious that their expenses were basically being used as no-interest loans by the company. The rate of exits accelerated after that.

What had been a thriving company became a dying, decaying shell of a company in under a year because of the management.

One Postscript:

One of the developers who left the old company ended up at the new company. He submitted his expense check. He was reasonable, he knew it would probably hit his next pay cycle. He was OK with that. I still recall the look on his face when later that day someone from finance walked in with his expense check. They were under no obligation to turn it around that fast and he certainly wasn’t expecting it. But they did so. They “bought” his loyalty that day by a simple gesture.

So, if people are leaving, trying to force them to stay will backfire. Figure out what you’re doing wrong and fix it.

Failure is Required

Posted on by
Reply

Last week one of my readers, Derek Lyons correctly called me out on some details on my post about Lock outs. Derek and I go back a long ways with a mutual interest in the space program. His background is in nuclear submarines and some of the details of operations and procedures he’s shared with me over the years have been of interest.  The US nuclear submarine program is built around “procedures” and since the adoption of their SUBSAFE program, has only suffered one hull-loss and that was with the non-SUBSAFE-certified USS Scorpion.

The space program is also well known for its heavy reliance on procedures and attention to detail and safety. Out of the Apollo 13 incident, we have the famous quote, “Failure is not an option” attributed to Gene Kranz in the movie (but there’s no record of him saying it at the time.)

Anyway, his comments got me thinking about failures in general.

And I’d argue that with certain activities and at a certain level, this is true. When it comes to bringing a crew home from the Moon, or launching nuclear missiles, or performing critical surgeries, failure is not an option.

But sometimes, not only is it an option I’d say it’s almost a requirement. I was reminded of this at a small event I was asked to help be a panelist at last week.  It turned out there were 3 of us panelists and just 2 students from a local program to help folks learn to code: AlbanyCanCode. The concept of agile development was brought up and the fact that agile development basically relies on failing fast and early.  For software development, the concept of failing fast really only costs you time. And agile proponents will argue that in fact it saves you time and money since you find your failures much earlier meaning you spend less time going down the wrong path.

But I’m going to shift gears here to an area that’s even more near and dear to my heart: cave rescue.  At an overarching, one might say strategic level, failure is not an option. We teach in the NCRC that our goal is to get the patient(s) out in as good or better shape than we found them as quickly and safely as possible.  In other words, if we end up killing a patient, but get them out really quickly, that’s considered a failure; whereas if we take twice as long, but get them out alive, that’s considered a success.

But how do we do that?  Where does failure come into play?

One of the first lessons I was taught by one of my mentors was to avoid “the mother of all discussions.” This lesson hit home during an incident in my Level 1 training here in New York. We had a mock patient in a Sked. Up to this point it had been walking passage through a stream with about 1″ of water. But we had hit a choke point where the main part of the ceiling came down to about 12″ above the floor passage.  There was alternative route that would involve lifting the patient up several feet and then over some boulders and through some narrow and low (but not 12″ low passage) and then we’d be back to walking passage.  I and two others were near the head of the litter.  At this point we had placed the litter on the ground (out of the water).  We scouted ahead to see how far the low passage went and noticed it went about a body length.  A very short distance.

Meanwhile the rest of our party were back in the larger passage having the mother of all discussions. They were discussing whether we should could drag the litter along the floor, lift it up to go high, or perhaps even for this part, remove the patient from the litter and have them drag themselves a bit.  There may have been other ideas too.

My two partners and I looked at each other, looked at the low passage, looked at the patient, shrugged our shoulders and dragged the patient through the low passage to the other side.

About 10 seconds later someone from the group having the mother of all discussions exclaimed, “where’s the patient?”

“Over here, we got him through, now can we move on?”

They crawled through and we completed the exercise.

So, our decision was a success. But what if it had been a failure. What if we realized that the patient’s nose was really 13″ higher than the floor in the 12″ passage. Simple, we’d have pulled the patient back out. Then we could have shut down the mother of all discussions and said, “we have to go high, we know for a fact the low passage won’t work.”

Failure here WAS an option and by actually TRYING something, we were able to quickly succeed or fail and move on to the next option.

Now obviously one has to use judgement here. What if the water filled passage was 14″ deep. Then no, my partners and I certainly would NOT have tried to move the patient with just the three of us. But perhaps we might have convinced the group to try.

The point is, sometimes it can often be faster and easier to actually attempt a concept than it is to discuss it to death and consider every possibility.

Time and time again I’ve seen students in our classes fall into the mother of all discussions rather than actually attempt something. If they actually attempt something they can learn very quickly if it will work or not. If it works, great, the discussion can now end and they can move on to the next challenge. If it doesn’t work, great, they’ve narrowed down their options and can discuss more intelligently about the remaining options (and then perhaps quickly iterate through those too.)

So today’s take away, is don’t be afraid of failure. Embrace it. Enjoy it. Experience it. It will lead to learning.  Just make sure you understand the price of failure.  Failure may be an option and is sometimes mandatory, but in other cases, the old saw is true, failure is not an option, especially if failure means the loss of life.