Copying a Large File

It was a pretty simple request actually. “Can you copy over the Panama database from FOO\WAS_21 to server BAR\LAX_45?”

“Sure, no problem.”

Of course it was a problem.  Here’s the issue. This is at one of my clients. They have a couple of datacenters and have hundreds of servers in each.  In addition, they have servers in different AD domains.  This helps them partition functionality and security requirements. Normally copying files between servers within a datacenter isn’t an issue. Even copying files between the different domains in the same datacenter isn’t normally too bad. To be clear, it’s not great.  Between servers in the same domain, it appears they have 1GB connections, between the domains, the firewall seems to throttle stuff down to 100MB.

The problem is when copying between different domains in different datacenters. This can be abysmally slow. That was my problem this week.  WAS_21 and LAX_45 were in different datacenters, and in different domains.

Now, for small files, I can use the cut and paste functionality built into RDP and simply cut and paste. This doesn’t work for large files. The file in this case was 19GB.  So this was out.

Fortunately, through the Citrix VDI they provide, I have a temp folder I can use. So, easily enough, I could copy the 19GB file from FOO\WAS_21 to that. That took just a few minutes.  Then I tried to copy it from there to BAR\LAX_45. This was slow, but looked like it would work.  It was going to take 4-5 hours, but they didn’t need the file for a week.

After about 4.5 hours, my RDP session locked up. I logged out and back in and saw the copy had failed. I tried again. This time at just under 4.5 hours I noticed an out of memory error. And then my session locked up.

So, apparently this wasn’t going to work. The obvious solution was to split the file (it was already compressed) into multiple files; except I’m not allowed to install most software on the servers. So that wasn’t a great option. I probably could have installed something like 7zip and then uninstalled it, but I didn’t want to deal with that and the paperwork that would result.

So I fell back to an old friend: Robocopy.  This appeared to be working great. Up until about 4.5 hours.  And guess what… another out of memory error.

But I LIKE challenges like this.

So I looked more closely. Robocopy has a lot of options. There are two that stuck out: /Z – restartable mode. That looked good. I figured worst case, I’d start my backup, let it fail at about 85% done and then resume it.

But then the holy grail: /J :: copy using unbuffered I/O (recommended for large files). 

Wow… unbuffered… that looks good. Might use less memory.

So I gambled and tried both.  And low and behold, 4:19 later… the file was copied!

So, it was an annoying problem but… I had solved it.  I like that!

So the take-away: Don’t give up. There’s always a way if you’re creative enough!

Math is Hard, Let’s Go Shopping

If I were to ask my readers to take a math test right now, approximately 1/2 would perform worse than if I had used a more neutral title such as “Math Quiz Below”. I’ll let you as a the reader guess which 1/2.

This is a subtle form of priming. Multiple studies have shown that by priming people before taking tests or making decisions, we can influence their outcome. It isn’t quite subliminal advertising, but it can be close.

I’m currently reading Delusions of Gender by Cordelia Fine and it’s quite the read. I recommend it to my audience here. She goes into the studies showing how priming can impact outcomes and references them in more detail.

Overall, we know that women are less represented in STEM fields, but this lack of representation doesn’t start out this way. Studies show in grade school the interest in STEM by gender is about equal. But over time, there’s less representation of women in most STEM fields and often when they are represented, their positions either carry less weight (not as much advancement) or perceived to carry less weight (ignored, spoken over, etc.) And before anyone comments, “but I know a woman who is a CTO at my company” or similar, keep in mind that those are noteworthy because they are the exceptions, not the norm.

Now, no single solution will solve the problem of women’s representation in STEM. But there are things we can do. First, we need to recognize that the human brain is probably built to be primed for certain responses. But don’t confuse this with saying that we can’t change what we’re primed for or how we respond. And, we can also avoid priming.

One study that is cited by Fine appears to suggest that collecting gender-biased demographic data AFTER a test or survey doesn’t cause a gender based result in the test. In other words, if you simply give a math test and then at the end ask questions like gender, or even to put ones name on it (which can often have a influence on self-perception) it appears to remove the bias towards poorer performance by women.  Similarly if you don’t ask at all.

But, most of us aren’t giving math tests are we?

But we are doing things like looking at resumes, deciding what conferences or seminars to attend, what blogs to read or respond it and how we interact with our coworkers and bosses.

One technique to consider is blind recruitment. Here much if not all demographic data is removed from a resume. This sort of work goes back to the Toronto Symphony Orchestra in the 1970s. But note, there is some evidence that it’s not the panacea some make it out to be. So proceed with caution.

When attending a conference or seminar, you can do one of a few things. For one, try to read the session descriptions without seeing the name of who is presenting. This can be a bit hard to do and may not quite get the results you want. Or, and I’m going to go out on a limb here because some people find this concept a bit sexist and I don’t have a great deal of data to support it, but…. go based on the names, and select sessions where woman are presenting. Yes, I’m suggesting making a conscious, some would say sexist, choice.

So far I’ve been pleasantly pleased by doing so. Over a year ago at SQL Saturday Chicago 2017 I decided to attend a session by Rie Irish called Let Her Finish: Supporting Women’s Voices from meetings to the board room. I’d like to say I was surprised to find that I was only 1 of 2 men in the room, but I wasn’t. I was a bit disappointed however, since really it was men who needed to hear the talk more than women.  Oh and the other gentleman, was a friend of Rie’s she had invited to attend. And a related tip, when attending such topics, generally, KEEP YOUR MOUTH SHUT. But that’s a different blog post for a different time.

Other great talks I’ve heard were Mindy Curnutt‘s talk at SQL Summit 2017 on Imposter Syndrome. Or Deborah Melkin’s Back to the Basics: T-SQL 101 at SQL Saturday Albany 2017. Despite her being a first time speaker and it being a 101 class, it was great and I learned some stuff and ended up inviting her to speak at our local user group in February of this year.

Besides making your fellow DBAs, SQL professionals, IT folks etc feel valuable and appreciated, you’re also showing the event coordinators that their selections were well made. If more people attend more sessions given by women, eventually there will be more women presenting simply because more will be asked to present.

But what if you can’t go?  Encourage others. Rie and her partner Kathi Kellenberger (whom I’m indebted to for encouraging me to write my first book) are the leaders of the PASS WIT (Women in Technology) Virtual Chapter of PASS. Generally before a SQL Saturday they will retweet announcements of the various women speaking. It doesn’t hurt for you to do the same, especially for women that you know and have heard speak.

But what about when there are no women, or they’re poorly represented. Call folks out on it. Within the past year we’ve seen a “Women in Math” poster, which featured no women.  There was a conference in Europe recently (I’m trying to find links) where women were extremely underrepresented. When women AND men finally started to speak out and threaten not to attend or speak, the conference seemingly suddenly found more women qualified to speak.

I’ve heard sometimes that “it’s hard to find women speakers” or “women don’t apply to speak”. The first is a sign of laziness. I can tell you right now, at least in the SQL world, it’s not hard. You just have to look around.  In the second case, there may be some truth to that. Sometimes you have to be more proactive in making sure that women are willing to apply and speak. For my SQL/PASS folks out there, I would suggest reaching to Rie and Kathi and finding out what you can do to help attract speakers to your conference or user group. Also, for example, if you don’t already have women speaking or in visible public positions within your organization, this can discourage women from applying because, rightly or wrongly, you’re giving off a signal that women may not be welcome.

Math may be hard, but it should not be because of gender bias, and we shouldn’t let gender bias, primed or not allow under-representation to occur.

PS – bonus points if anyone can recognize the mountains in the photo at the top.

PPS – Some of the links below may end up outdated but:

And that’s just a small sampling of who is out there!

 

Alarming

So a recent trip to the ER (no, nothing serious, wasn’t me, thanks for asking) reminded me of a topic near and dear to my heart: Alarms and Alerts. What prompted this thought was the number of beeps, boops, and chirps I heard while there that no one responded to.  This leads to the question: Why have them, if no one responds to them?

I have a simple rule for alarms: “Don’t put an alert on something unless you have a response pre-planned for it.”

This is actually more complex than it sounds. And it can sometimes lead to seemingly illogical conclusions if you follow it in a reductio ad absurdum fashion.

Let’s start with an example of one alert I heard while sitting waiting. It was a constant beep, about 90 times a minute. I soon tracked it down to a portable monitor attached to a patient that was soon to be moved upstairs.  It was the person’s pulse.  Besides a possible HIPAA violation (I was now in theory privy to private medical information) it really served no purpose other than to annoy the patient and those around them. “But Greg, perhaps they were afraid the patient would suddenly go into cardiac arrest or something else would happen.”  And I agree, but then let’s alert on the sudden change in conditions, not in what was, at the time, a stable pulse for the patient. This beeping went on for over 10 minutes. And no one was monitoring it, other than the patient and us annoyed strangers.

So, there was an alert, that apparently needed no response.

But let’s go to the other extreme. What about when an alert isn’t needed. Let’s say you’re driving your car and it throws a rod. (Yes, this happened to me once, well I wasn’t driving, my father was. It was his sister’s Volkswagon campervan). I can tell you there is NO alert when such an event happens. But, there’s no need for it. The vehicle stops. It won’t go. So an alert in that case is pretty superfluous.

But let’s tie this to IT. I’m going to give you an absurd example of when not to have an alert: When you run out of disk space.  Again, you might disagree. You’d think this would be the perfect time to have an alert. But go back to my rule. What if you have no plan for this? You’ve never gamed out the possibility.  Now, you’re out of disk space. You don’t have a plan. Does it really matter if you had an alert or not? If you can’t respond, the alert really hasn’t added anything.

The main lesson to take away from that example is, if you’re setting up an alert, make sure you do have a plan. (The other lesson of course is perhaps to have an alert BEFORE you run out of disk space!) The plan may be as simple as, “delete as many files as I can”. But of course that only works if you have files to delete. Or it might be “add another filegroup to the database for now and then figure out the long-term solution during our next planned outage.”  Or, in the worst case it might be, “update my resume.”  But the point is, if you have an alert, have SOME plan for it.

On the flip side, how many times do you have an alert that you look at and say, “oh yeah, we can ignore that, that always happens.”  Sure, that’s a plan, but honestly, ask yourself, do you need an alert in that case? Probably not. I hate getting woken up at 2:00 AM for an alert I don’t need to respond to.  So in this case if there is no plan because you don’t need a plan, eliminate the alert.

I could go on (and perhaps this will be a good topic for my next book) but I’ll add one last real-world case where people all to often ignore alerts: smoke and CO detectors; especially CO detectors.  If you have a CO detector and it alerts, do NOT assume it’s faulty and unplug it. Respond. Somehow. Don’t automatically assume it’s a faulty battery, especially if it’s the winter. If you have any doubt, please call the fire department. Trust me, they’d much rather respond to a call where you’re all alive and it’s a false CO alarm than to show up and find the alarm going off, but everyone is now dead.

So the take away is, alerts are only useful if they generate a useful response.

Oh and because the inner child in me can’t resist: be a lert because the world needs more lerts! 🙂

 

 

Wet Paint

There’s an old saw that if you tell someone there’s 1 billion people in China, they’ll believe you, but if you put up a wet paint sign, they’ll have to touch the paint to be sure. I find it interesting that we believe some things easily and others not so easily.  Or even sometimes, we may think we believe something until we actually experience it. Then we somehow believe it even more.

Two incidents of this nature occurred to me within the last year.  Last year I drove to my uncle’s house in South Carolina in order to observe the total eclipse. Prior to totality, one of the phenomena that I knew would happen was seeing crescent shaped “shadows”. (Really it’s sort of the reverse since you’re seeing crescent shaped light.) I mean I had read about it, I had seen pictures, and I basically understood why they would occur.  And yet, at one point I was walking back into the house to do something when I looked down and lo and behold… I saw crescent shaped “shadows”.  My reaction was one of “Holy Cow, this really DOES happen!”  Now, I had no doubt intellectually that it should happen, or that I would most likely see it. But, actually seeing it was still amazing and while I really had no reason to have to verify the phenomenon, the fact that I personally had experienced it was incredible. I understood it at a visceral level, not just an intellectual one.

The second incident occurred just the other day. As a long-time sufferer of allergies, spring time has sometimes been a bit less than comfortable for me. And over the years, I had seen videos of pine trees releasing their pollen in a massive cloud (I’m thankfully NOT allergic to pine pollen it appears) but I had never actually experienced it myself. So again, I knew intellectually it happens.

So two days ago I was sitting outside looking at a pine tree when suddenly I saw this puff of what looked like smoke, and then a cloud of pollen waft away from the tree. Again, I had experienced that Holy Cow moment when the visceral experience matched the intellectual one. It was pretty cool.

That all said, I have stopped touching wet paint at this point in my life. But I still love these sort of confirming experiences (though I’m not eager to start counting heads in China at this time).

What facts did you know that when you finally experienced them first hand, had an impact upon you. I’d love to hear them.

Teamwork

I spent the majority of last week with some of the greatest people I know: my fellow NCRC (National Cave Rescue Commission) instructors and students.  Let’s start by the obvious, it takes a strange twist of mind to be the sort of person that actually enjoys crawling through dark holes in the ground. Now take that same group of people and add  a sense of altruism and you’ve got folks willing to rescue others trapped in caves.

Let me interject in here by the way, that caving is actually a fantastically safe sport. When an accident in a cave makes the news, it’s because it’s so rare and generally so unique as to attract attention. The NSS puts out a report every 2 years detailing the accidents across the US. It’s well worth the read if you can find a copy. Now back to the rest of my blog post.

I have been one of the many instructors who teach the “Level 2” class. I love this level because students have gotten beyond the basics and are starting to learn the “why” we do things more than simply the “how”.  And we start to really work on their team leadership and teamwork skills (as I mentioned last week in my post about the lost Sked).

Good teamwork isn’t just a bunch of people working to solve a task. It’s them working together and often anticipating the needs.  Two events last week illustrated a failure and a success.  At one point, I was a mock patient and the class had been broken into two teams. The first one had to get me out of a tight crawl-way (packaged in a Sked) and up to an open area. From there another team had set up a haul system to get me to the top of a 60′ or so (I’ve never really measured it) block of rock.  This was towards the end of the week when the students (and instructors) are a bit tired and overwhelmed with everything they had learned.  We had allocated 90 minutes for the exercise.

During the first part, I just felt like something was off. Nothing serious, nothing I could put my finger on. But the magic the students had shown all week just wasn’t quite there.

And then there I was, 2 hours into the exercise, laying on top of the tight area, waiting to be hauled up. Someone on the lower team said that I was ready to go.  Someone at the top of the block said they were ready to go.  And then I sat for 2-3 minutes until finally things started happening again.  But for a few minutes the days of teamwork had fallen apart. They weren’t trying to anticipate needs or even work cooperatively.  It wasn’t a huge issue, but it did get a reaction from our oldest, most irascible Level 2 instructor.  I think the word ‘disappointed’ was used at least twice.  It might have been a bit harsh, but… it worked.

As the final exercise of the day, the instructors decided to have a bit of fun with the students. We hid the previously lost (and then found) Sked.  The instructor in charge then informed the students that their lost “patient” for this particular exercise was about 4′ tall, last seen wearing bright orange, and was afraid of the dark. As details were added, it suddenly dawned on them what we had in mind.

But they took the task seriously. They did an excellent search and when they were then told the “patient” wasn’t responsive to stimuli, and they couldn’t rule out a c-spine injury, therefore the Sked had to be packaged in another litter, they did so with gusto and honestly, one of the best packaging jobs I had seen all week. All this while laughing.

They took an absolutely silly scenario, laughed while doing it, yet exhibited amazing teamwork. They were back on their game!

Sometimes teams can start to fall apart, but reminding them of how good they can be and providing a bit of levity can help elevate them back to their best!

 

A Lost Sked

Not much time to write this week. I’m off in Alabama crawling around in the bowels of the Earth teaching cave rescue to a bunch of enthusiastic students. The level I teach focuses on teamwork. And sometimes you find teams forming in the most interesting ways.

Yesterday our focus was on some activities in a cave (this one known as Pettyjohn’s) that included a type of a litter known as a Sked. When packaged it’s about 9″ in diameter and 4′ tall. It’s packaged in a bright orange carrier. It’s hard to miss.

And yet, at dinner, the students were a bit frantic; they could not account for the Sked. After some discussion they determined it was most likely left in the cave.

As an instructor, I wasn’t overly concerned, I figured it would be found and if not, it’s part of the reason our organization has a budget for lost or broken equipment, even if it’s expensive.

That said, what was quite reassuring was that the students completely gelled as a team. There was no finger pointing, no casting blame. Instead, they figured out a plan, determined who would go back to look for it and when. In the end, the Sked was found and everyone was happy.

The moral is, sometimes an incident like this can turn into a group of individuals who are blaming everyone else, or it can turn a group into a team where everyone is sharing responsibility. In this case it was it was the latter and I’m quite pleased.

Legacy

“…the good is oft interred with their bones.”

Or these days, lives on in the Internet. I never quite agreed with Shakespeare in this line. I think the good lives on beyond the grave.

In the book, Lies my Teacher Taught Me author James Loewen talks about how certain African tribes divide people into three categories: those alive, the sasha or living-dead, and zamani or the dead.

This was brought to mind yesterday when I was trying to debug an issue which turned out to be a bug in SQL Server 2016 SP2.  While trying to debug it, I needed to add a user to an SSIS setup. This has been a problem in the past, but I recalled I had used #SQLHELP on Twitter to ask the question and gotten a great answer. So, a quick search later found the response I was looking for. The fully correct answer (since MSFT’s page leaves out a step) was available at: http://sqlsoldier.net/wp/sqlserver/howdoigrantaccesspermissionsforssistousers

Now, many of my readers won’t recognize the name, but some will: @SQLSoldier, a member of the #SQLFamily that passed away recently. At the time of his passing I had forgotten that he had reached out to help me last year. The search yesterday though brought it back to me. I never had the honor of meeting Mr. Davis in person, but I know many others spoke highly of him. It was comforting to me to know that even months later his legacy was still helping me (and presumably others).

After thinking about that, I got thinking about my dad.  Soon after he passed in 2015, I picked up the hefty Milwaukee right-angle drill that had been his and was now mine. I was working on the addition (that he had helped design before his death) that has since become my office. I had always liked that particular tool. It has a certain heft and power to it.  At the time, with his death so close at hand, it was a form of grief therapy for me. I had often used this in my youth, helping him out with various construction projects. To this day I’ll pick up one of the tools I inherited, or start a house project using the skills he taught me and I realize, he’s sasha, living-dead. He’s still lives on in me.

SQLSoldier is also living-dead, he’s very real in the hearts and minds of those who knew him and his legacy lives on, still helping others, such as myself.

As my age is now entering its 2nd half-century,  I wonder more and more what my legacy will be. I hope that when I’m sasha, my legacy can still help and aid others.

And with that, I will conclude with a scene from one of my favorite actors in one of my favorite movies:  “What will your verse be?”