Giving Blood and Pride Month

I gave blood yesterday. It got me thinking. First, let me show a few screenshots:

male blood donor shot 1

7 Male Donor #1 screen shot

female blood donor shot 1

Female Donor #1 screen shot

Let me interject here I’m using the terms Male and Female based on the criteria I selected in the American Red Cross’s Fast Pass screen. More on why I make that distinction further on. But first two more screen shots.

female blood donor shot 2

Pregnancy question highlighted for female

male blood donor shot 2

No pregnancy question for males

Now, on the face of it, this second set of questions especially almost seems to make sense: I mean if I answered Male early on in the questionnaire, why by asked about a pregnancy? But what I’m asked at the beginning is about my gender, not my actual child-bearing capability. Let me quote from Merriam-Webster:

2-b: the behavioral, cultural, or psychological traits typically associated with one sex

Or from the World Health Organization:

Gender refers to the roles, behaviours, activities, attributes and opportunities that any society considers appropriate for girls and boys, and women and men. Gender interacts with, but is different from, the binary categories of biological sex.

Who can be pregnant?

So above, really what the Red Cross is asking isn’t about my gender, but really my ability to be pregnant. Now, this is a valid medical concern. There are risks they want to avoid in regards to pregnant women, or recently pregnant women giving blood. So their ultimate goal isn’t the problem, but their initial assumption might be. A trans-man might still be able to get pregnant, and a trans-woman might be incapable of getting pregnant (as well as a cis-woman might be incapable.) And this is why I had the caveat above about using the terms male and female. I’m using the terms provided which may not be the most accurate.

Assumptions on risk factors

The first set of images is a problematic in another way: it is making assumptions about risk factors. Now, I think we can all agree that keeping blood borne pathogens such as HIV out of the blood supply is a good one. And yes, while donated blood is tested, it can be even safer if people who know they are HIV or at risk for it can potentially self-select themselves out of the donation process.

But…

Let me show the actual question:

Male Male 3 month contact question

Question 21, for Men

This is an improvement over the older restrictions that were at one year and at one point “any time since 1977”. Think about that. If a man had had sex with another man in 1986, but consistently tested negative for HIV/AIDS for the following 30+ years, they could not give blood under previous rules. By the way, I will make a note here that these rules are NOT set by the American Red Cross, but rather by the FDA. So don’t get too angry at the Red Cross for this.

The argument for a 3 month window apparently was based on the fact that HIV tests now are good enough that they can pick up viral particles after that window (i.e. at say 2 months, you may be infected, but the tests may not detect it.)

Based on the CDC information I found today, in 2018, male-to-male sexual contact resulted in 24,933 new infections. The 2nd highest category was heterosexual contact (note the CDC page doesn’t seem to specify the word sexual there.) So yes, statistically it appears male-male sexual contact is a high-risk category.

But…

I know a number of gay and bisexual men. I don’t inquire about their sexual habits. However, a number are either married or appear to be in monogamous relationships. This means if they want to give blood and not lie on the forms, they have to be celibate for at least 3 months at a time!  But hey if you’re a straight guy and had sex with 4 different women in the last week, no problem, as long as you didn’t pay any of them for sex! I’ll add that more than one gay man I know wants to give blood and based on their actual behavior are in a low risk category, but can’t because of the above question.

Why do I bring all this up at the end of Pride Month and what, if anything does it have to do with database design (something I do try to actually write about from time to time)?

As a cis-het male (assigned at birth and still fits me) it’s easy to be oblivious to the problematic nature of the questions on such an innocuous and arguably well-intended  form. The FDA has certain mandates that the Red Cross (and other blood donation agencies) must follow. And I think the mandates are often well-intended. But, there are probably better ways of approaching the goals, in the examples given above, of helping to rule out higher-risk donations. I’ll be honest, I’m not always sure the best way.  To some extent, it might be as simple as rewording the question. In others, it might be necessary to redesign the database to better reflect the realities of gender and sex, after all bits are cheap.

But I want to tie this into something I’ve said before: diversity in hiring is critical and I think we in the data world need to be aware of this. There are several reasons, but I want to focus on one for now.

Our Databases Model the World as We Know It.

The way we build databases is an attempt to model the world. If we are only aware of two genders, we will build our databases to reflect this. But sometimes we have to stop and ask, “do we even need to ask that question?” For one thing, we potentially add the issue of having to deal with Personally Identifiable Information that we don’t really need.  For another, we can make assumptions: “Oh they’re male, they can’t get pregnant so this drug won’t be an issue.”

Now, I’m fortunate enough to have a number of friends who fall into various places on the LGBTQIA+ (and constantly growing collection of letters) panoply and the more I listen, the more complexity I see in the world and how we record it.

This is not to say that you must go out instantly and hire 20 different DBAs, each representing a different identity. That’s obviously not practical. But, I suspect if your staff is made up of cis-het men, your data models may be suffering and you may not even be aware of it!

So, listen to others when they talk about their experiences, do research, get to know more people with experiences and genders and sexualities different from yours. You’ll learn something and you also might build databases. But more importantly, you’ll get to know some great people and become a better person yourself. Trust me on that.

 

 

 

The Customer is Always Right?

You’ve heard this adage before. Some often believe it. And honestly, there’s a bit of truth to it. But the truth is, the customer pays your bills and if they stop paying your bills, they’re no longer your customer.

I was reminded of this actually during the testimony of Dr. Fauci before the Senate a few weeks ago. To be clear, neither the Senate nor the CDC is a customer here, nor is the President of the United States. But, I think it ties into the thesis I want to make.

Over the past few months there’s been a lot of discourse over whether states should shut down, for how long and how and when they should open up. At the extreme ends you have folks who seem to argue for a continual shutdown to save as many lives as possible and the people who seem to argue that the economy is far more important and that any shutdown is a bad idea.

Dr. Fauci has been accused of wanting to “quarantine the entire country” and is the subject of a hashtag campaign, #faucifraud. During the Senate hearing, Senator Rand Paul took several swipes at Dr. Fauci including a statement implying that some people might treat Dr. Fauci as the end-all. Finally, with only 32 seconds left, Dr. Fauci gave his reply:

“Well, first of all, Senator Paul, thank you for your comments. I have never made myself out to be the end all and only voice in this. I’m a scientist, a physician, and a public health official. I give advice, according to the best scientific evidence. There are a number of other people who come into that and give advice that are more related to the things that you spoke about, about the need to get the country back open again, and economically. I don’t give advice about economic things. I don’t get advice about anything other than public health. So I wanted to respond to that.”

I think this was an incredible reply and one that I think behooves any consultant to keep in mind. Dr. Fauci politely but firmly refutes Senator Paul’s comment about being the end all and then points out what his qualifications are. He then suggests that there are other experts, in other fields, who should be consulted. He then reiterates the limitations of his advice.

As a consultant, this mirrors my own experience. A client may ask me to recommend a HA/DR strategy for them. I might go ahead and recommend some sort of Always On Availability Group with multiple synchronous replicas in one data center and then an asynchronous replica to a second data center. I might then recommend daily backups to tape with the tapes taken off-site. Everything would be encrypted and we would test failovers on a regular basis. With that, the proper selection of hardware, a proper deployment setup, and a completely developed runbook for various scenarios, I could probably guarantee them nearly perfect uptime.

Then, the CFO steps in and points out that their budget is only 1/20th of what I had planned around and that trying to spend more would bankrupt the company.

Then the VP of Sales points out that the business model of the company is such that in reality, they could operate for several hours of downtime and while it might hurt business a little bit, it wouldn’t bring them to a halt.  In fact, they suggest that the order system just be done with a bunch of Excel spreadsheets that accounting can true up at the end of the month. After all, they just want to focus on sales, not on entering data into the system.

Finally the CEO steps in. They decide that it’s true, the company can’t afford a 24/7 HA/DR setup that is the envy of NASA, at least not at this time. Nor do they think the VP of Sales plan has much merit since it won’t allow future growth into online sales and while it might be easier for the salespeople to just jot down stuff, it would mean hiring more people in accounting to figure out the data at the end of each month.

Instead, they direct the CTO to work with all the parties involved to develop a system that can have 3 hours of downtime, but costs no more than 1/10th of what I proposed, and  that also incorporates features that allow them to move to a more advanced HA/DR setup down the road and also will eventually allow for online sales.

So who was right? Me at my extreme of a huge investment in hardware, licenses and resources, or the VP of Sales who wanted to do the whole thing using some Excel spreadsheets.

Both and neither. Either of our ways would have worked, but neither was the best solution for the company.

I think good experts realize exactly where their expertise begins and ends. My role as a consultant is to provide the best advice I can to a company and hope that they take my advice, at least as it is applicable. And I should understand that every situation is different. In these cases, the customer is always right. Their final decision might not be what recommended, but ideally they’ve taken it into consideration.

Finally though, I have to recognize that there are situations where I may have to withdraw myself from the situation. In the above scenario, I crafted a situation where compromise is not only a viable option, it is perhaps the best option. But there are times when compromise is not an option. If a potential client came to me and they were dealing with PII data and refused all advice in regards to encrypting data and other forms of data security, it would be in my best interests to simply say, “here, the customer is wrong” and recuse myself. So in some cases, the customer may be outright wrong, but they should also stop being my customer at that point and would no longer be paying me.

So, do we re-open the country completely or do we shutdown everything until the fall? Honestly, I’m glad I’m not the one making that decision. I don’t think there’s a single answer for every community. But I think the best leaders will take into account the best advice they can from a variety of experts and synthesize the best answer that they can and adjust it as more data and experience come to light. It’s simply not practical to prevent every possible COVID-19 death. But it’s also not ethical to re-open without a plan or even thought as to the impact.

Neither extreme is fully right.

 

Advanced Braining

I’m currently reading the tome The Power Broker by Robert Caro. For those not familiar with it, it’s the Pulitzer Prize winning biography of Robert Moses. “Robert who?” you may be asking? Robert Moses, perhaps more than any single person literally shaped New York City in the mid-20th Century. Due to his power, he was responsible in NYC alone, for getting the Triborough Bridge, Brooklyn-Battery Tunnel, West Side Highway, Cross-Bronx Expressway, and many other large scale projects built. He outlived a number of borough presidents, mayors, governors and even Presidents. Arguably, for decades he was the most powerful man in NYC, at least in terms of how many was spent and what projects were completed. In many ways he was a visionary.

However, as the chapter I’m currently in discusses, he also could be extremely short-sighted. I’ll come back to that in a moment.

In the past week, several small incidents occurred in my life. Separately, they don’t necessarily mean much, but taken together, I realized there was a little theme associated with them.

Last Tuesday I posted an update on my dryer repair and an issue at one of my clients. I described the work incident as an example of the normalization of deviance. A few hours later, someone I’ve known for decades, originally online, but have since met in person, Derek Lyons (who has a great blog of his own on anime, a subject about which I know nothing) posted a reply to me on Facebook and said he had read my article, liked it, but thought I was wrong. I was intrigued. You can see his comment and my reply at the bottom of last week’s post. The general point though is I think he showed my thinking was incomplete, or at least my explanation was. In either case, it made the overall article a better one.

Then on Wednesday, my editor at Redgate, Kathi Kellenberger  emailed me with changes to my most recently submitted article. One of the changes was to the title of the article. Now, I’ve come to value Kathi’s input, but I wasn’t keen on the title change, so I suggested something different. She wrote back and recommended we go with hers, How to Add Help to PowerShell Scripts because she said “How to…” generates more hits on search engines and in fact a previous article of mine How to Use Parameters in PowerShell was one of their most read articles at the time (106K hits and climbing). I went with her advice.

Yesterday, a friend contacted me. He was in the middle of doing grading for his students and the numbers on his Excel spreadsheet weren’t quite making sense. The errors weren’t huge, but just enough to make him go “hmmm”. So, he reached out to me to take a look. After a few minutes of digging I understood what was happening and able to write back to him and give him a better solution.

All these have something in common: the final product was better because of collaboration. This is a common theme of mine: I’ve talked about the chat system I use at RPI, I’ve talked about making mistakes. In general, I think that when trying to solve a problem, getting additional input is often valuable.

So back to Robert Moses. In the early part of his career, before his efforts focused mostly on NYC itself, he was responsible for other projects, such as the Northern State Parkway and the Southern State Parkway and Jones Beach on Long Island. He started his career in a time when cars were mostly a vehicle of the well-off and driving a parkway was expected to be a pleasant experience (hence the name). His efforts were built around more and more parkways and highways.

By the 1950s though, it was becoming apparent to most everyone else that additional highways actually generated more traffic than they routed away from the area surface roads. What was originally considered a blessing in disguise, where a bridge, such as the Triborough would quickly generate more traffic (and hence more tolls) than expected, was soon seen as a curse. For every bridge or tunnel built in or around NYC, traffic increased far more than expected. And this came at a price. Urban planners around the country were starting to see the effects. Efforts to build more bridges or highways to ease traffic congestion were actually creating more. Even in NYC as Moses was planning for his next large projects opposition was slowly building. However, Robert Moses was blind to the problem. By the 1950s and 60s he had so surrounded himself by “yes men” that no dissidence was permitted. In addition, opposition outside of this offices was silenced by almost any means Moses could use, including apparently the use of private detectives to dig up dirt on opponents.

In the current chapter I’m reading, Caro, the author, details exactly how much money the Triborough Bridge Authority (which was in practice, though not theory, under Moses absolute control) and the Port Authority had available for upcoming projects, including the planned Verrazzano-Narrows Bridge. He goes on to explain how badly the infrastructure of the NY Subway system and the LIRR had fallen into disrepair. Caro suggests how much better things could have been had just a portion of the money the TBA and PA had at their authority had been spent on things like the Second Avenue Subway (something that is only now coming to fruition and will take possibly decades more to complete). Part of the issues with the subway system can be lain directly at the feet of Moses due to earlier efforts of his to get the city to fund his other projects. The issues with the LIRR however were more an indirect result of his highway building out into Long Island.

I suspect some of Caro’s claims are a bit idealistic and would have cost more than the projections at the time (like most projects) and while I think most of the projects he touches upon probably should have been built in the 50s (the Second Avenue Subway being one of them and the LIRR East Side Access being another) they weren’t because of a single man who brooked no disagreement and was unwilling to reconsider his plans.

Robert Moses was a man who got things done. Oftentimes that’s a good thing. And honestly, I think a number of his achievements are remarkable and worthy of praise.

But I have to wonder, how much better of a city could New York be, had Robert Moses listened to others, especially in the 1950s and 60s.

Today’s takeaway? Take the time to listen to input and ask for it. You may end up with a better solution in the long wrong.

 

 

 

Bits are cheap

And, as unfortunately as a recent incident in our #SQLFamily community illustrated, apparently at times so is respect.  Bear with me as I relate these two ideas and another incident.

Let me start with a statement that should make more sense by the end of this post: My name is Gregory, but I prefer that you call me Greg. My pronouns are he/him/his.

But first a trip down memory lane. Many of us recall the Y2K issue. This was a direct result of programmers decades ago trying to save bytes in storage (and to a lesser extent memory and CPU cycles) because storage was expensive. By storing dates as just the last two digits of the year, they could cut the storage for years in half. This was important back then because it saved money. But, as many of us recall, as the year 2000 approached, this started to cause more and more problems. (As a point aside, the first example I’m aware of was brought to my attention by a programmer who worked for a bank in 1970. Seems as if they suddenly had issues handling 30 year mortgages!)

Since then of course the cost of storage has dropped and as an industry we’ve moved to storing years as a 4-digit year. No one in today’s day and age would normally question this decision.

But enough of ancient history, let me get to the point of this article: respecting others.

As many readers know, those of us on Twitter will often use the hashtag #SQLFamily.  In the past week I’ve seen two incidents that have illustrated the worst and the best of this family.

In the first case, a member of the community, a woman I had never met, said she was leaving the family, she no longer felt welcome. At an event she had been misgendered not once, but multiple times. For those who aren’t sure what that means, I will, without going into background or details (because they’re not important) say she is a trans-woman. Several people at the event took it upon themselves to refer to her using by male pronouns.

In the most recent case, a fellow speaker, Cathrine Wilhemsen tweeted about how she had been addressed as Cathi and Kathi twice in the previous 24 hours. She says this hasn’t been the only time, but just the most recent and recent enough for her to comment on.

In both cases, part of the problem is that strangers addressed the person in question in a manner that did not respect them; in the first case by not using the proper pronouns and in the second by not using her provided name.

But that’s one part of the problem.  So let’s address that: we have members of the #sqlfamily who don’t respect other members. But, we have another issue, and one that I think is important to address: those who minimize the issue. In the first case, apparently no one called out the folks misgendering the woman.  In a situation like this, a show of support can be as simple as saying something like, “Umm, I think you mean she, not he.”  You can also support the use of pronouns on nametags at events or in the bio descriptions for events.

Remember though, today, bits are cheap. So we can do more. Don’t design your database with a bit field for gender. Make it a table. These are relational databases after all. Have a table for possible gender identifications. Allow for a method to add rows to this table. Have a table for pronouns.  There’s more than you might think and people are often crafting additional ones. While the singular they/them is becoming more popular, it’s NOT the only alternative to he/him, she/hers.

We are data professionals after all. We absolutely should not lock our data into a single view of the world if that worldview is changing. (Note, the world is not changing, there have been multiple genders throughout recorded history.  We’re simply becoming more cognizant of it now.)

In the case of Cathrine being called by another name, keep it simple. Use the name provided, be it in an introduction, on the nametag or other method. Respect the person’s wishes. And do not, as some did on Twitter respond by “well they probably didn’t mean anything” or “eh, just roll with it.” It’s not YOUR name. It’s not YOUR identity. Sure, you might not care if someone calls you Richard, Rick, Ricky or Dick. But another person might. Their name is part of their identity, respect their wishes.  I will add one more note that Cathrine shared with me and that other women have shared with me, it is almost always men that will use nicknames or cute names or similar without prompting.  Yes, fellow men, I’m calling you out. We may not think about it. In fact I would argue we often don’t think about it. It’s something that privilege allows us. But be aware that your attempt to be friendly or familiar is actually often coming off as diminishing and condescending.

Now, despite the failure of some members of #SQLFamily, I want to celebrate the great people in the community. These two incidents have created a lot of responses. I’ve seen at least two great posts, one from Jen McCown and another from Kellyn Gorman. I’m sure there are others. I also have written in the past about being an ally. But in addition, while I’ve seen one or two tweets that have dismissed Cathrine’s tweet, I’ve seen many members rally to the defense of the women in both incidents. And, also very importantly, I’ve seen several tweets from people asking, “how can I help?” or “how can I improve my behavior?” I love that last one. I’m constantly trying to unlearn some of the behaviors I was taught and to be more conscious of what being a white, straight cis-het male brings to the table. We can always learn to do better.

Yes, our #SQLFamily has some members who could and need to do better. That saddens me. Fortunately as I’ve seen, it also has a lot of members actively striving to do better and help others do better. That gladdens me. Let’s all be the latter.

Respect and disk space don’t cost us much. Let’s learn to be respectful of people and to design databases that can also respect the world around us.

P.S. I want to note, I was purposely vague about the first incident because the specifics weren’t important and I did not want to draw more attention to a specific person without their permission. In Cathrine’s case, I made a point of respecting her and exchanged messages with her first to make sure she was ok with me bringing more attention to the incident.

This Site Makes Cookies

Apparently under new guidelines here and in Europe I’m ethically obligated that I’ve been known to make cookies from time to time.  Oh, excuse me, something is coming in to my earpiece now.  Oh, never mind, I’ve been informed those laws apply to a different type of cookie.

In any event, I first got into the habit of baking cookies on a somewhat regular basis while in college. It became a stress release for me, and also apparently made me quite popular among the sorority sisters and outing club members I lived with.  I would, probably at least once a month my sophomore year make a double-batch of Tollhouse Chocolate Chip cookies. They rarely lasted more than a day or two.

Since then, I’ve expanded my repertoire, including once trying “bacon cookies” for my very first SQL Saturday. Those weren’t a huge hit, but haven’t stopped me from baking.

That said, I’ve learned a few things over the years about baking cookies. For example, my daughter would bring cookies to school for an event and would often be asked, “oh did your mom make them?”  She’d patiently explain that no, her dad did. Even today, the assumption is that when it comes to school events, the mom does the baking. I’m glad that my kids both realize that it’s unfair to expect that mothers have to do all the baking and other domestic duties.

But, I also learned something else that sort of threw me for a loop. People don’t like homemade cookies from a zip-lock bag.  Sometimes I’d bring cookies to events and people wouldn’t eat many of them. Now, being practical and in a hurry, I’d almost always just toss the cookies into a zip-lock bag.  It was my daughter who suggested I start putting them into a plastic container with a lid instead. Suddenly I found the same cookies were much more popular. My daughter explained her theory, which I tend to believe. For whatever reason, perhaps hygiene, people don’t want to reach into plastic bags for food. It may be touching the same sides that everyone else did or something else. But regardless, putting them into plastic tub with a container works.

Call it a UI problem, but, it seems to work.

Today’s take-away, just because you’re comfortable with a solution and think it works, don’t be adverse to making changes, even if they seem silly or trivial, if that’s what your users desire.

P.S.: Check out my latest writing for Red-Gate: PowerShell and Secure Strings.

Two Minds are Better Than One

I’m going to do something often not seen in social media. I’m going to talk about a mistake that I made. It’s all to common on various certain media sites to talk about how perfect our lives are and how great things are. You rarely hear about mistakes. I decided, in the theme of this blog of talking about how we approach and solve (or don’t solve) problems, I’d be up front and admit a mistake.

This all started with a leak in the downstairs shower. It had been growing over the years and I frankly had been ignoring it.  Why put off to tomorrow what you can put off to next month or even year? But finally, in December of last year it became obvious that it was time to fix the leak. It had continued to grow, and now that my son was home from college for extended break, he had setup a work area in the basement, below the bathroom.  I figured he didn’t really need to suffer from water dripping onto his desk.

So, he and I went into full demolition mode and ripped out the old tile and backer board to get to the plumbing.

New plumbing in bathroom

New plumbing

You can see some of the work here. I also took the opportunity to run wiring to finally put in a bathroom fan. That’s a whole other story.

Anyway, the demolition and plumbing went well. Then we put up the backer board and sealed it. And left it like that. It didn’t look good, but it was waterproof and usable. It was “good enough”. So for about 8 months it sat like that. But with an upcoming pool party, I decided it was time to finally finish it off. One of the hold ups had been deciding on tile. Fortunately, on a shopping trip about a month earlier, my son, my wife and I found tile we liked (my daughter, who ironically still lives at home and will be using the shower more in the next few years than her brother, was in LA on vacation, so she ended up not really having much say in the matter).

So, earlier this month, while the rest of my family took a weekend to go to Six Flags New Jersey, I figured I’d surprise them with finally tiling the shower.  I went to the big box store whose favorite color is orange and bought the required materials. Since tile we had selected is approximately 6″ wide and 24″ long, I had to make sure I got the right mastic.  This was a bit different from stuff I’ve worked with in the past with a bit more synthetic materials in it and it mixed differently, and had slightly different drying characteristics. That, combined with growing darkness lead me to move quickly. The darkness was a factor since any tile cutting I was doing was outside.

And, all that lead to a simple mistake.  On the end wall, there’s a window and as a result part of that wall needed tiles just less than 24″ long. On one hand, this is a huge plus since it means less seams and less places to grout. On the other, it meant in one spot having to work around the trim of the window. And that’s where I made my mistake.  The trim had been put over the original tile, so there was in theory room behind the stool of the window to fit in a piece of tile. That had been my plan.

But, when it came to sliding in the nearly 24″ long piece of tile, it wouldn’t fit. It wouldn’t bend (obviously) to let me get it tucked behind the stool and due to the stickiness of the mastic, I couldn’t slid it in from the top.

So, I cut out a notch. In the back of my mind I somehow was thinking, that it wouldn’t look that bad and tile would cover it.  Well, I was obviously wrong.

In hindsight, I realized I should have cut the tile in two pieces, created an extra seam (like the row below it that had to cover more than 24″ wide in any event) and then I could have slide in the smaller piece and then put in the remaining piece. It would have been perfect, looked great and more likely to be waterproof.

So, this gets me to the title. Had I been doing the work with someone else, I’m sure I’d have said, “Damn, this is gonna suck, any ideas?”

A mistake

My mistake

And I’m sure someone else would have suggested, “Hey, cut the piece and slide it in.”

I like working alone, but sometimes, you need a second person to help. Or more than one brain as I’ve mentioned in the past.  If nothing else, sometimes a duck can help.

Idera Ducks!

A bunch of rubber ducks, including two from Idera!

So, the moral of the story is sometimes two heads are better than one. Oh well, I won’t be the one using that shower, so I won’t see my mistake all the time!

Oh and check out my latest Red-Gate Article on Secure Strings in PowerShell.

Security: Close isn’t good enough!

I was going to write about something else and just happened to see a tweet from Grant Fritchey that prompted a change in topics.

I’ve written in the past about good and bad password and security polices. And yes, often bad security can be worse than no security, but generally no security is the worst option of all.

Grant’s comment reminded me of two incidents I’ve been involved with over the years that didn’t end well for others.

In the first case, during the first dot-com bubble, I was asked to partake in the due diligence of a company we were looking to acquire. I expected to spend a lot of time on the project, but literally spent about 30 minutes before I sent an email saying it wasn’t worth going further.

Like all dot-com companies, they had a website. That is after all, sort of a requirement to be a dot-com. And it was obvious it was backed by a database server (which I knew was SQL Server, which sped up my process, but only by a few minutes). So, I did the obvious thing and got the IP address of the web-site. Then, I simply tried to connect to SQL Server from my desktop to that IP address and one or two on either side of it. On my second attempt, the IP address right before the one of the website replied to my attempt to reach SQL Server. That was not a good sign. The reply meant there was no effective firewall in place. Note, had they not been using SQL Server, but some other tech, it might have taken me another 10-15 minutes to find the right client to connect. So knowing it was SQL Server wasn’t overly important.

But of course at least they had a password right? Well, back then, the latest and greatest version of SQL Server was 2000 which still did not require a password when you set it up.  I asked myself, “it couldn’t be that easy could it?”

Sure enough it was. Within minutes I had logged in as sa without a password. I now had complete control of their SQL Server. But even more so, back then SQL Server allowed unfettered access to xp_cmdshell. In theory at that point I could have done anything I wanted on the box, including installing remote access software and creating and giving myself administrator access.  I didn’t. But, my email to my boss was short and sweet. I explained how there was absolutely no way we could acquire their platform without a complete top to bottom review of it for any signs of malware. If it took me only 30 minutes or less to get in, I was almost certain their system was owned.

We never acquired that company. I’ve wondered since then what happened to them. My guess is, like many dot-com companies they folded. I can’t say it would have been because of their lack of security, but I can say that the lack of security played a huge factor in us NOT acquiring them. (and for the record, the company I worked for at the time ended up acquiring 1-2 other companies, merging with a 3rd and finally being acquired by a 4th, which is still around. So we were doing something mostly right.)

The second incident that comes to mind was about 8 years later at another start-up. I was asked by the COO to do some due diligence on the setup in another division’s datacenter setup. Again, I didn’t do anything fancy. I knew they weren’t running SQL Server, but I figured I could still do some probing. This time what I found was a bit different. It wasn’t software per se, but rather their iSCSI switch. Sure enough not only did it have a public facing IP address, but, the CTO of that division had failed to change the default password. I was very tempted at the time to give the IP address to my 8 year old son, without any other details and asking him to try to log in. Given his skills, even at that age, I’m 99% sure he’d have figured how to Google the required information and get in. But I figured I didn’t really need to do that to make my point.

That and other factors later lead to the CTO leaving the company.

Moral of the story: Make sure your sensitive information is under some form of lock and key and don’t use blank or factory default passwords, let you or your company end up in a headline like this one: Evisort Data Exposed.

Design Thoughts

Ever look at a product and wonder, “why did they design it that way?” I know I have, and I have some examples I want to bring up.

Years ago, over dinner, I had a programmer from our Wisconsin office basically ask, “why the hell is your file system for your web servers setup the way it is?”  It was a fair question. It wasn’t something one would normally see.  But before I explain that…

Like any modern American, I’m physically incapable of being more than 10′ from a flat screen TV in my house.  We have several, including one in my office and one in the kitchen. I couldn’t tell you the brand of the one in the kitchen (well I could, but I’m too lazy to go downstairs and find out) and the only reason I can tell you the brand of the one in my office is because I can see it from here. It’s an Inginsia brand.

Both serve the same function: they allow me to watch TV. But both have design quirks.

Their button layouts are a bit different (note the layout of the numbers and the volume/channel control buttons.)

Kitchen TV Remote

Kitchen TV Remote

Office TV Remote

Office TV Remote

The kitchen TV also has a built-in DVD player, so it has additional controls for that.

So obviously, there’s different design philosophies and requirements here. But I want to go a step deeper and talk a bit about functionality.

The kitchen TV remote, if you mistype a number, you can hit the Vol – button and it will essentially backspace and delete the number. Actually a handy feature.  The Office remote has no such functionality, though hitting EXIT will remove the entire channel already entered.  Score one for Dynex. (Ok, I did go downstairs so I could grab the remote and take a photo).

But, the Dynex has one annoying quirk I’ve never figured out. When I hit the OFF button, there’s a noticeable delay of 1-2 seconds before it actually turns off. For the life of me, I have NO idea why. I mean I’m turning off a TV. It’s not like I’m shutting down a computer where it has to write the contents of memory to disk and perform other tasks. Sure, maybe it has to save the last channel I was tuned to, but it could do that right after I tuned into that station. Same with the volume.  Every other TV in the house, including my office one, when you hit the off button, turns off instantly.

I’m reminded a bit of early computers that had the big red switch. There was something satisfying about turning off an early PC. You knew it was instantly off. There was no two questions about it. Now, shutting off a PC is a far more complex operation and can take sometime.  But a TV? I’d love to know why the kitchen TV takes a long time to turn off.

Now back to the file design the programmer was asking me about. Essentially we had 5-6 web front ends, each with a virtual directory in IIS pointing to a NAS. Not an entirely awful setup, but uncommon at the time.  We were offering a web platform to newspapers so they could publish their content. Originally we tried using a 3rd party package to make sure the content on all the servers was always in synch (since a newspaper could upload content at any time to any of the servers and wanted it available instantly). What we found was sometimes we’d get into race conditions where files could actually end up erasing themselves. The 3rd party company kept assuring us they had the solution. Well after a desperate call at 4:00 AM call from my on-duty NOC person, I drove into the office, scrambling to figure out a better solution. On the drive, the idea of using the virtual directories to the NAS occurred to me. We implemented it in about 30 minutes and solved our problems. It was supposed to be a temporary solution until we came up with a more robust, permanent solution. But, 18 months later it was still in place, working great and I was explaining it to our out of town programmer. He went from, “that’s nuts” to “Hey, that makes a lot of sense.”

So, I like to think that when there’s a design I don’t understand, the designers at the time had their reasons. But, to be honest, I’m not always sure.

For example, the photo that should be heading up this article, of a shampoo bottle and a bottle of conditioner, both from the same manufacturer, both designed to be cap down, are printed the opposite way. The only reason I can think of that makes sense is so that in a befuddled, sleep deprived state, I can more easily determine which is which. But even if that is the answer, why this way, and not the other? Inquiring minds want to know!

And yes, the shampoo bottle can be placed cap up, but the conditioner bottle can’t be. Again, why? The viscosity of the two aren’t that different. Again, inquiring minds want to know.

Shampoo/Conditioner bottles

One of these is upside down!

 

Fail-safes

Dam it Jim, I’m a Doctor, not a civil engineer

I grew up near a small hydro-electric dam in CT. I was fascinated by it (and still am in many ways). One of the details I found interesting was that on top of this concrete structure they had what I later found are often called flashboards. These were 2x8s (perhaps a bit wider) running the length of the top of the dam, held in place by wooden supports.  The general idea was they increased the pooling depth by 8″ or so, but in the advent of a very heavy water flow or flood, they could be easily removed (in many cases removed simply by the force of the water itself).  They safely provided more water, but were designed in fact to fail (i.e. give away) in a safe and predictable manner.

This is an important detail that some designers of systems often don’t think about; how to fail. They spend so much time trying to PREVENT a failure, they don’t think about how the system will react in the EVENT of a failure. Properly designed systems assume that at some point failure IS an not only an option, it’s inevitable.

When I was first taught rigging for cave rescue, we were always taught “Have a mainline and a belay”.  The assumption is, that the system may fail. So, we spent a lot of time learning how to design a good belay system. The thinking has changed a bit these days, often we’re as likely to have TWO “mainlines” and switch between them, but the general concept is still the same, in the event of a failure EITHER line should be able to catch the load safely and be able to recover. (i.e. simply catching the fall but not being able to resume operations is insufficient.)

So, your systems. Do you think about failures and recovery?

Let me tell you about the one that prompted this post.  Years ago, for a client I built a log-shipping backup system for them. It uses SSH and other tools to get the files from their office to the corporate datacenter.  Because of the network setup, I can’t use the built-in SQL Server log-shipping copy commands.

But that’s not the real point. The real point is… “stuff happens”. Sometimes the network connection dies. Sometimes the copy hangs, or they reboot the server in the office in the middle of a copy, etc. Basically “things break”.

And, there’s another problem I have NOT been able to fix, that only started about 2 years ago (so for about 5 years it was not a problem.) Basically the SQL Server in the datacenter starts to have a memory leak and applying the log-files fails and I start to get errors.

Now, I HATE error emails. When this system fails, I can easily get like 60 an hour (every database, 4 times an hour plus a few other error emails). That’s annoying.

AND it was costing the customer every time I had to go in and fix things.

So, on the receiving side I setup a job to restart SQL Server and Agent every 12 hours (if we ever go into production we’ll have to solve the memory leak, but at this time we’ve decided it’s such a low priority as to not bother, and since it’s related to the log-shipping and if we failed over we’d be turning off log-shipping, it’s considered even less of an issue). This job comes in handy a bit later in the story.

Now, on the SENDING side, as I’ve said, sometimes the network would fail, they’d reboot in the middle of a copy or something random would make the copy job get “stuck”. This meant rather than simply failing, it would keep running, but not doing anything.

So, I eventually enabled a “deadman’s switch” in this job. If it runs for more than 12 hours, it will kill itself so that it can run normally again at the next scheduled time.

Now, here’s what often happens. The job will get stuck. I’ll start to get email alerts from the datacenter that it has been too long since logfiles have been applied. I’ll go in to the office server, kill the job and then manually run it. Then I’ll go into the datacenter, and make sure the jobs there are running.  It works and doesn’t take long. But, it takes time and I have to charge the customer.

So, this weekend…

the job on the office server got stuck. So I decided to test my failsafes/deadman switches.

I turned off SQL Agent in the datacenter, knowing that later that night my “cycle” job would turn it back on. This was simply so I wouldn’t get flooded with emails.

And, I left the stuck job in the office as is. I wanted to confirm the deadman’s switch would kick in and kill it and then restart it.

Sure enough later that day, the log files started flowing to the datacenter as expected.

Then a few hours later the SQL Agent in the datacenter started up again and log-shipping picked up where it left off.

So, basically I had an end to end test that when something breaks, on either end, the system can recover without human intervention. That’s pretty reassuring. I like knowing it’s that robust.

Failures Happen

And in this case… I’ve tested the system and it can handle them. That lets me sleep better at night.

Can your systems handle failure robustly?

 

 

Hours for the week

Like I say, I don’t generally post SQL specific stuff because, well there’s so many blogs out there that do. But what the heck.

Had a problem the other day. I needed to return the hours worked per timerange for a specific employee. And if they worked no hours, return 0.  So basically had to deal with gaps.

There’s lots of solutions out there, this is mine:

Alter procedure GetEmployeeHoursByDate @startdate date, @enddate date , @userID varchar(25)
as

— Usage exec GetEmployeeHoursByDate ‘2018-01-07’, ‘2018-01-13’, ‘gmoore’

— Author: Greg D. Moore
— Date: 2018-02-12
— Version: 1.0

— Get the totals for the days in question

 

 

set NOCOUNT on

— First let’s create simple table that just has the range of dates we want

; WITH daterange AS (
SELECT @startdate AS WorkDate
UNION ALL
SELECT DATEADD(dd, 1, WorkDate)
FROM daterange s
WHERE DATEADD(dd, 1, WorkDate) <= @enddate)

 

select dr.workdate as workdate, coalesce(a.dailyhours,0) as DailyHours from
(
— Here we get the hours worked and sum them up for that person.

select ph.WorkDate, sum(ph.Hours) as DailyHours from ProjectHours ph
where ph.UserID=@userid
and ph.workdate>= @startdate and ph.workdate <= @enddate
group by ph.workdate
) as a
right outer join daterange dr on dr.WorkDate=a.WorkDate — now join our table of dates to our hours and put in 0 for dates we don’t have hours for
order by workdate

GO

There’s probably better ways, but this worked for me. What’s your solution?