Page 81

Posted on by
Reply

One of the things that has always fascinated me is human memory; how we create them, what sticks, what doesn’t and how it evolves.

Many people think that human memories are rather static. The truth is that’s far from the truth.  As we saw in the 1990s is easy to form false memories and easy to conflate them.

One detail that is interesting is that human memories are a bit like DRAM in a computer.  In essence when we recall a memory, we have to basically read out the memory space and write it back.  One of the side effects is this can actually help strengthen memories.  However, it also means when it’s written back, other memories can be conflated with it and a new, slightly different memory is formed.

There’s two main ways of remember something that stand out to me as I write this.  Repetition and what I’ll call “sudden shock”.

Many things we need to repeat until we remember them.  An example is a child learning their times tables. There’s really not much context and really only rote repetition will cause these to sink in.

At the other end of the spectrum are the memories that are etched in our minds. “Where were you when Challenger blew up?”  “How did you first hear about 9/11?”  If you ask someone of the right age, they’ll know exactly when/where they were and probably recall vivid details.

If you ask them where they were on the 3rd shuttle mission, they’d probably have no clue.  The same is true if you ask them what they were doing on 9/9.

In between are more general memories. Memories of childhood that don’t necessarily have a specific timestamp or even importance.  I recall playing in some woods behind my house growing up, but there was nothing really significant about the time or place. I have no idea why I have that memory.

I mentioned above that memories can be modified or manipulated. There’s some work on treating PTSD this way; helping patients recall specific events under controlled circumstances and essentially rewriting the memory into something that doesn’t cause an attack. (Propanolol is one drug being experimented with to do this.)

Strangely there’s one memory of mine that persists that while not a real issue is sort of pointless and annoying to me.  It’s “Page 81”.

What’s that you ask? Many years ago (let’s just say before I was a teenager I think) I was staying at my cousin’s grandmother’s house.  On the bookshelf they had a copy of Jaws 2. I started reading it but had to leave before I could finish it. Since I knew I’d be back the next summer I decided to remember what page I was on. I repeated the page number to myself over and over again. And to this day, I can remember, I was on page 81 of Jaws 2 when I stopped reading.  Of course decades later I have no idea what happened in pages 1-80 so the memory doesn’t do me much good. But there it is. It’s still there. Page 81.

As a note, most of this post was based on memory (I had to look up the name of the drug) so some details may be wrong.

Page 81.

 

 

On Call

Posted on by
Reply

I want to pass on a video I’ve finally gotten around to watching:

Dave O’Conner speaks

I’ve managed a number of on-call teams to various levels of success. One point I’d add that makes a difference is good buy-in from above.

He addresses several good points, most of which I would fully agree with and even at various times adopted at my various jobs.

One thing he mentions is availability.  Too often folks claim they need 99.999% uptime. My question has often been “why?” and then followed by, “Are you willing to pay for that?”  Often the why boils down to “umm.. because…” and the paying for it was “no”, at least once they realized the true cost.

I also had a rule that I sometimes used: “If there was no possible response or no response necessary, don’t bother alerting!”.

An example might be traffic flow.  I’ve seen setups where if the traffic exceeds a certain threshold once in say a one hour period (assume monitoring every 5 seconds) a page would go out.  Why? By the time you respond it’s gone and there’s nothing to do.

A far better response is to automate it such that if it happens more than X times in Y minutes, THEN send an alert.

In some cases, simply retrying works.  In the SQL world I’ve seen re-index jobs fail due to locking or other issues.  I like my sleep.  So I set up most of my jobs to retry at least once on failure.

Then, later I’ll review the logs. If I see constant issue of retries I’ll schedule time to fix it.

At one client, we had an issue where a job would randomly fail maybe once a month.  They would page someone about it, who would rerun the job and it would succeed.

I looked at the history and realized simply by putting a delay in of about 5 minutes on a failure and retrying would reduce the number of times someone had to be called from about once a month to once every 3 years or so.  Fifteen minutes of reviewing the problem during a normal 9-5 timeframe and 5 minutes of checking the math and implementing the fix meant the on-call person could get more sleep every month. A real win.

Moral of the story: Not every thing is critical and if it is, handle it as if it is, not as a second thought.

American Cave Accidents

Posted on by
1

I’ve mentioned in the past that I teach cave rescue and I work with some great people.  This week I received the biannual American Cave Accidents report that the NSS puts out. (I’ll mention I’m friends with the editor and several of the folks who helped advise on this issue.)

This report in theory covers all reported cave accidents in the last 2 years in the US (and sometimes Canada and Mexico and other parts of the Americas.) In reality, anyone who is familiar with caving knows it misses some.

As I was reading today, I noticed in  several of the reports, the names of several people that I recognized; most are fellow instructors, or at the very least students.

Now, one could apply correlation is causation and determine that perhaps the presence of NCRC trained people causes caving accidents.  Thankfully I’m pretty sure that’s not true. But it got me thinking, “Why does it seem such a high proportion of the accident reports involve people I know?”  I certainly do NOT know that many cavers.

Then it dawned on me: these people are perhaps far more likely to actually submit accident reports. I know of at least two accidents where the people involved did NOT want to report.  Why? Because they thought either it was a bit embarrassing or because they didn’t think there was anything to learn from their experience.

I posit those who go through the NCRC training realize more than many others the value of sharing their experiences and that it is perhaps it is worth risking a little embarrassment (and to be honest there never really is any embarrassment in my opinion) if others can learn from their experiences.

I happened to be talking to another friend of mine tonight who is also in the EMS field and he also brought up as a separate discussion how much value there was in shared knowledge like this and how some folks don’t like to do so.

The more you can broaden your horizons and experiences you can hear about and share in your field of expertise will make you a better person in that field.

Take the time to share your mistakes and your success. Share your unusual stories, you never know when someone else can benefit from them.

Sharing and teaching

Posted on by
4

I spent this past Saturday in NYC at another SQL Saturday event. This blog isn’t typically about my day job. But sometimes things overlap.

SQL Saturday is basically a community run, volunteer event where folks who use SQL Server, or are interested in it, get together, present and attend talks and share ideas.

I unfortunately missed the keynote given by Grant Fritchey, but from what I understand, he discussed the importance of networking for DBAs: and he didn’t mean the kind running on TCP/IP handshakes as much as the one running on human handshakes.

This is exactly why I’ve wanted to teach at a SQL Saturday for awhile. I can’t claim my presentation was as well attended as many others and I can’t claim my presentation was as useful as some, but it was well received. (I would be lying if I didn’t say I think handing out cookies didn’t help!)

I’m a firm believer in passing on what I’ve learned. It’s how I learned, others took the time to pass information on to me and I take the time to pass it on to others.

One thing I really like about the SQL Saturday community is how much folks leverage off of each other. Often an idea presented by one presenter is seized upon by another who then expands upon the idea and adds to the body of knowledge.  Then the first presenter will take the updated idea even further.

This is how we learn, by taking ideas, expanding and sharing them back.

Never run out of a plan

Posted on by
1

I’ve actually been meaning to blog about this for awhile, but have been putting it off, so here goes.

I’ve mentioned in the past my analogy of “flying the plan”. Lately I’ve been spending a lot of time on a site called Quora. It’s quite a fun site and I’ve learned quite a bit.

But this particular question I think is a great one for life in general.

Scrolling down, you’ll see a post from Jim Mantle. I want to take a quote from his answer:

There have been many air crashes where a problem was being worked by both pilots, neither was flying the aircraft, and they had a Very Bad Day.

If you read about the L1011 Crash you’ll see the real mistake was failing to actually fly the plane. The crew was so engrossed in solving the problem of a burnt-out landing gear light that they missed the fact that the plane was flying into the ground.  A simple burned out bulb and 101 people died.

Compare that to the Miracle on the Hudson where the pilots had a MUCH worse problem (lack of power in either engine) and managed to bring the plane down safely without any loss of life.

He also has good advice that he repeats often “Keep calm”.

I also want to quote Dirk Van Der Walk who later says:

You can run out of height, you can run out of engine, but one thing you can never run out of, is a plan. You must always have a Plan B.

I had a client a few years ago that had called me in to implement a specific change in their infrastructure.  There was also a fairly specific timetable by which it had to be done.

I met with CTO about once a month to go over the status of the project.  At one point it became clear that due to certain corporate policies, it would take about 12 weeks to get to a certain milestone in the project.  Unfortunately the schedule demanded we be there in about 8 weeks.

He asked me what we could do.  I explained I had no control over the corporate policies and that we should start to consider a Plan B.  I’m quite proud that I kept my jaw from hitting the floor when he uttered his next sentence.

This is no plan B and there can’t be a plan B.

This is an example of taking the mantra “Failure is not an option” to a whole new level.

Ironically I was there about a month later when the CTO was basically called out on the carpet for the status of the project and when it was clear he had no plan B, the corporate folks spent the next 24 hours designing a plan B.

In part this wasn’t too hard because the internal people on the project had already had several plan B’s in their mind.

It was only because others did have a plan B that we were able to save any real semblance of the original goal.

Moral of the story: always have a backup plan.  And start thinking about a backup plan to the backup plan.

Rolling in the Deep

Posted on by
Reply

I was at SQL Saturday in Boston this past weekend and I sat in on a session given by Paresh Motiwala: “Why do we shun using tools for DBA job?”

It’s a decent question and sometimes rolling your own (hence the title here and yes I’m listening to Adele right now) is the right answer.

But often, it’s not.

A case in point are DBAs who avoid using the built-in SQL Server maintenance plans, especially for simple tasks such as Backups.

Now, I’ll start out by saying straight up, sometimes they’re not the optimal solution. (You may want to backup certain partitions on a different a rotation schedule and the like as one example.)

Below is a recent situation I came across.

But, if you do decide to roll your own, please do NOT do the following:

  1. First write a sproc that has to be placed into each database as it’s created in order to run the backup. (If you DO decide to go this route, please make this a scripted part of your release procedure so it’s not missed.)
  2. If you DO in fact ignore suggestion #1, I will point out that it doesn’t do much good to make the first step of your sproc to check to see if the database exists. BIG HINT: If the database doesn’t exist, the sproc won’t run in the first place! Yes, I know you’re saying, “Obviously” but this is the situation I just came across.
  3. If you DO in fact ignore suggestion #2, you probably don’t need to call your own special “alert sproc”.
  4. But if you DO in fact ignore suggestion #3, make sure your “alert sproc” does more than call a “email sproc” and pass it a few parameters.
  5. And if you DO in fact ignore suggestion #4, please make sure your “email sproc” does a bit more than build a message and call the built-in SQL Server stored proc to send an email.

Now granted, there may be reasons to do some of the above. Perhaps your alert sproc also calls some sort of 3rd party monitoring tool. Or your email needs are very specific.

In the case I just rectified, none of that was true.  So there was a lot of additional complexity (which really didn’t work anyway) for no good reason.

Another problem this roll your own backup setup had was that it used the same filename every time for its backups. i.e. On Monday the backup name was M:\Backups\FOO_FULL.BAK.  On Tuesday it was M:\Backups\FOO_FULL.BAK, etc.

In theory (and generally in practice) each of these would be backed-up to a 3rd party so it was in theory possible to find the backup for a specific day, but that was an added complexity; and probably not one you want in an actual DR situation.  Also, if for some reason the backup to the the 3rd party failed (but the local backups continued) they’d definitely lose the ability to restore specific days of backups.

In addition, the person who built this procedure setup differential backups for MOST databases to run every 15 minutes.  Now, I wouldn’t necessarily call that a terrible idea, but in this case, almost certainly not the best approach in my opinion.  However, again, the same file name was used each time.

This means that in a DR event, the company could restore the previous nights backup and if they wanted, the most recent Diff backup and that was it.  If they wanted to restore to a point in time in between, that was impossible. And in my experience this is far more common than most other restore needs.

Finally, the developer who wrote all this clearly did not understand what the CHECKPOINT command did. He had scheduled a checkpoint job to run every 30 minutes.  Again, in general, not only not necessary, but probably a bad idea. However in this case it not only was not necessary, the reason given in the job comments was completely wrong.  He seemed to think it would keep the transaction logs from growing.

This is of course NOT what it does and sure enough on the one database still with FULL RECOVERY enabled the transaction log was far larger than the actual database. (Fortunately it was a lightly used database or the disk might have filled up years ago.)

Since discovering all this, I’ve gone and replaced all this complexity with a set of maintenance jobs.  These will guarantee each system database is backed up weekly (for their needs this should be fine) with unique names.  User databases will be backed up nightly and retained for 4 nights (and perhaps extended once we determine fully how much disk space we want to set aside for this.)  Transaction logs will be performed every 15 minutes. These too will have unique names.

Now the customer can restore to any point in time in the last 4 days (if they go to their 3rd party backup, even further back) up to 15 minutes before a failure (and in some cases if the log is still available and they can backup the tail of the log, up to the instant before the failure).

If they add additional databases, they don’t have to worry about remembering to put in 3 separate sprocs for each database added and adding new jobs to the SQL Server Agent.

Now they not only have a far more robust backup plan, they have one that is far easier to maintain.  Oh and one that will actually send an email if there’s a problem with the backup.

The morale is: Don’t make things more complex unless you absolutely need to and if you do, make sure you actually achieve the goals you’re trying to achieve.

Getting the right answer by suggesting the wrong one

Posted on by
Reply

I’m a participant on a CMC called Lily It is based out of my alma mater, RPI.  At some point, someone created a rule (which I’ve seen elsewhere so it’s hardly unique) that sometimes the fastest way to get the right answer to a question is to post the wrong answer.

There is truth to that.  I think in part it can be summed up with this XKCD cartoon.  Many of us who are involved in technology seem to have an incessant need to be “right”.  So when we see something wrong, we’re compelled to correct the mistake.

But, to be wrong, it has to be clearly wrong.  To go back to my cave rescue experience, if I recommend a 3:1 haul system and you recommend a 2:1, neither of us is necessarily wrong. We might be optimizing for different factors.  On the other hand, if you recommend we use 11mm rope for the haul line and I whip out some clothesline I’ve had in my car for a few years and suggest it should be good enough, after all it’s only Bill we’re rescuing, I’m clearly going to be wrong and need to be corrected.

These thoughts about being wrong and trying to find the right answer were prompted by a coding problem that has consumed far too much of my time. I finally came up with an answer that worked, but not one that I liked.

Essentially I’m building a Combobox (loading it from a datatable) in vb.net

It has key,value pairs, let’s call them (“Test1”, “A”), (“Test2”, “B”) and so forth.
(note VB.net appears to call these a DisplayMember,ValueMember pair and they can be loaded with a dictionary type, so in my mind it’s what they call the “valuemember” is what I’d consider the lookup key and that illustrate my misunderstanding of the issue.)

However, once I load the record in question, I want the selected value in the dropdown to reflect the value in the record (which of course is stored as “A” or “B” etc.)

There appears to be no way in VB.Net to easily say something like:

cbxResource.SelectedValue = Itemrecord.Value

Then I tried:

cbxResource.SelectedItem = Itemrecord.Item just to see if it would work. It doesn’t.

Googling suggests something like:

cbxResource.SelectedIndex = cbxResource.FindString(Itemrecord.Item)

That does indeed work, if I know the DisplayMember name. But that’s I want to display, not what I store in Itemrecord and as such means I don’t know it.

It strangely seems I can not set the index based on the ValueMember, just the DisplayMember.  To me this is strange since coming from a DB world, it appears the value member would be the key I’d want to look  up to select the Displaymember to be displayed.

I finally settled on a hack.  What if I switched the two?

cbxResources.DisplayMember = “Resource”
cbxResources.ValueMember = “Description”

cbxResources.SelectedIndex = cbxResources.FindStringExact(Itemrecord.Item)

cbxResources.DisplayMember = “Description”
cbxResources.ValueMember = “Resource”

I’m not sure I like this answer. It seems to me it should be far simpler. Or that I’m fundamentally misunderstanding how the control should be setup and used.  But for now it’s the hack that’s going into my code.

So why publish here?  Well either it’s a great work-around and I can save other folks the hours of fruitless searching I experienced, or someone can say, “It’s on the Internet and it’s wrong; I have to correct it!”

I’ll take either answer.

Moral: Sometimes being wrong is the right thing to do.

Practicing for Disaster

Posted on by
Reply

I’ve had this post by Wayne Hale in my queue for awhile since I’ve wanted to comment on it for awhile and until lately have been to busy to do so.

One of my current contracts requires them to do an annual DR test.  Since the end of the year is approaching, they’re trying to get the test in. Part of the test requires an “official” test observed by an outside auditor.

So, being smart, and since a lot has changed in the past year, we decide to schedule a dry-run or two before hand.

Well let’s just say those have not gone as expected.

Some might consider the dry-runs failures.

I don’t. I consider them successes. We are finding out now, in a controlled environment with no real time pressures, where we are weak and need to fix things.

It’s far better to do this now than during the audited test or even better than during an actual disaster event! So the dry-runs are serving their purpose, they’re helping us find the holes before it’s too late.

That said, I have to claim the part that I’m most involved with, the SQL Log-Shipping has been working well.  The only issue this week with that was a human error made by another DBA that was completely unrelated to the DR test and within minutes of him discovering his error he executed the proper procedure to begin fixing it.  The total fix on his end took no more than 5 minutes and other than monitoring on my end, the effort on my end took no more than 5 minutes.  That’s an excellent example of a robust design and set of procedures.

Today’s moral is don’t just have a DR plan, practice it. And not every failure is really a failure.

Shiny Things

Posted on by
Reply

It’s not uncommon for fads or “shiny things” to rise to the top.  In my work in cave rescue for example, there’s a lot of really cool equipment that comes out every year.  And often a particular piece or type of equipment will grab peoples attention and folks will start to use it all over the place. A good example of this are various “rope grabs” such as a Rescucender. Suddenly everyone is teaching, “Oh, you should use this tool, it’s so much better.” In part people fall into the trap of thinking it’s better because it’s newer.

Another example are ascending systems.  For climbing ropes in caves there are Frogs, Ropewalkers, Texas and more.  They all have their advantages. And quite honestly, for the most part, they are far better than what they replaced, the traditional “3-knot” system.  Unfortunately, as an instructor I’ve come across a large number of experienced vertical cavers who have NO idea how to climb on a 3-knot system.

Rescucenders have their place and they had their peak of prominence. For awhile it seemed they were used and taught all over the place. A lot of teams I know are going back to focusing on a good old Pruisk Knot for a rope grab.  This of course is the most common knot used in 3-knot climbing systems.

In one case, the newer, better, shinier thing was found not necessarily to be all that better.  In the other case, the newer equipment really is better. But the basics shouldn’t be forgotten. You never know when you’ll need to fashion Prusik’s out of shoelaces.

I bring this up because of an article brought to my attention today by , discussing SQL vs. NoSQL. Many have told me that NoSQL will replace a traditional SQL RDBMS. Of course I was told the same thing about OODBMS and other database systems. So far many of the replacements have withered on the vine and are long gone.  Some still thrive in niche applications.  I don’t think NoSQL will wither and die; nor is it simply a small niche.  Yet neither will it completely replace SQL databases either.  Both have incredible powers and are very good at what they do.  Both I think are here to stay for the foreseeable future.

But don’t fall into the trap of thinking you have a data problem and assuming the newer solution, NoSQL is the better solution.  It might be.  It might not. If I have to climb rope, I’ll use the “newer” (it’s decades old now) technology of my Frog climbing system. If I need to put a rope-grab on a rope, I’ll almost certainly go back to the “older” (it’s all relative) technology of a Prusik knot.

The actual needs of the application drive the solution. Don’t allow the solution to drive the design.

A good rule of thumb is: Anytime anyone sets up a debate, especially about technology as an X vs. Y debate, they’re probably framing it wrong from the get-go. It’s rarely X xor Y, but almost always X and/or Y, or even possibly Z.

All set to think

Posted on by
Reply

A short post today.

This past weekend I attended another SQL Saturday event. This one in Washington DC.  For anyone interested in SQL Server, I highly recommend these events. Typically the cost is just $10 to attend for the day.

I had put in a bid to present but unfortunately was declined.  That’s fine, there were plenty of other seminars worth my time to attend.

One of them, Common Coding Mistakes and how to Mitigate Them by William Wolf was a good example of that. @SQLWareWolf did a great job of illustrating a number of fairly common mistakes.  That itself was worth the price of admission. But what I really enjoyed was an observation of his, one that I’ve had in the past.  As a DBA, I can often spot stored procs written by a programmer with a non-SQL background.  The tell-tale is that many (certainly not all) programmers who come to SQL from another background often think in terms of rows, not sets.

An example of this would be the case of a programmer opening up a cursor and looping through the cursor to set the date on a number of records.  Of course for any person with a SQL background, we understand that can be a single statement which will execute far faster.

Now, there are certainly times when row by row is the only way to do it, but if you see that in a sproc or script, I’ll bet you 5:1 it was written by a programmer who didn’t know better.

Now, to be fair as a DBA who learned to program when “object oriented” wasn’t a buzz-word, I’ve been doing a lot of VB.net programming lately and I have to admit often I’ll find myself down a hole of twisty paths before I’ll realize I’m writing bad code and if I simply think of the data I’m trying to modify as an object, suddenly things get FAR more clear and easier.  I would not be surprised if 5 years from now the next programmer comes along, looks at my code and thinks, “What was he thinking? He obviously understood objects, but didn’t do everything he should have using objects.”

But in both cases, over time the non-SQL programmer coming to SQL will learn and I hope my OO code is slowly getting better.

No real moral here other than common observations and that hopefully overtime, we all improve.