The Year So Far

Today happens to be the last day of the month and the last day of the quarter. And according to my calendar, it’s the 4th Blursberyday of the month of Holiecouw.

I decided to take a look back at my first post of the year: 2020 in Preview. Wow, a lot has changed in a scant three months. I mentioned I was reading Station Eleven. It’s set in a post-apocalyptic world after a world-wide flu pandemic. Little did I know at the time I’d be living that reality a scant 3 months later. Ok, this is not nearly as bad as in the book, but it does give on pause to think. We are living in a time of upheaval and it will be interesting to see how this current pandemic changes social structures for coming years.

I wanted to speak at SQL Saturdays. Well, almost every one I’ve put in for or was planning on putting in for has been cancelled or delayed. So much for that goal. On the other hand, members of the #SQLFamily have been holding Friday afternoon (and other times) Zoom hangouts as sort of a morale boost. So I’ve actually gotten to know a number of my fellow DBAs and fellow speakers, so that’s better.

Fortunately, I’m still working. As a consultant, you realize every meal may be your last meal, so you keep working at it and hoping more meals are coming your way. So far my biggest client shows no sign of slowing down, nor does my second largest client. I’ve been fortunate, I know a number of folks across many industries who have been hit with a temporary or even permanent job loss. This is going to be hard for many.

But, I’ve also been taking the time to do more webinars. Last week I sat in on a Redgate webinar on the state of DevOps that was quite informative. The next day, Kendra Little (also of Redgate) gave the WIT webinar and also talked about DevOps. Both were quite informative and I learned a lot. I look forward to the upcoming Redgate Streamed event.

I’ve been using git more and more. I started using it integrated with Visual Studio about two years ago I think. But, after seeing my son working on a project where he was using it at the command line, I decided it was time to start to do that and now for one client that’s my de facto way of checking in and out changes I’ve been making to the PowerShell scripts I write for them. Next up, more version control for the SQL Scripts. I’ve already written a small deploy script I use to deploy scripts and changes and more importantly to log them. So while that client hasn’t really adopted DevOps, I’m doing my part for my small corner of work.

My next goal is probably starting to learn how to use Docker more. Cathrine Wilhemson’s blog post on that has convinced me it’s time.

And I finally finished binge-watching Haven.

So, the last few weeks haven’t been exactly what I planned for, and the upcoming months won’t be what I planned on either, but it hasn’t been a terrible time. What about you?

P.S. While out biking the other day, a thought dawned on me. Many post-apocalyptic books (such as Station Eleven) have characters using cars, but more like carts, either pulling them themselves or with horses because once the gas runs out, you can’t make more. But I got wondering how having a large number of electric vehicles would play out in such a world. Yes, much of the infrastructure would be gone, but even if you had to carry panels with you (much like Mark Watney in The Martian) you could probably be far more mobile. Hmm…

It’s Easier the Other Way!

This is what someone said to me while biking up a hill the other day.

I have a certain bike route I do that includes stopping at the supermarket along the way. It’s just over 5 miles. Obviously over the course of the entire route I return to the same place I started, but the topography varies along the way. But the simple fact of the matter is that the supermarket is at a lower altitude than the house, and about 2/3rds of the way along the route in the direction I go.

So when I tell people about biking this route, I point out that it’s sort of a double-whammy. When I get to the lower point in the route, I go shopping. To finish my loop, my bike is often heavier with groceries and I have all the altitude to gain back (which admittedly isn’t much, a bit over a 100′) in a short distance (maybe 2000′).  It was along this section that my erstwhile adviser offered his advice.  My reply of course was that “true, but home is that way!”, I said pointing up the hill.

Unfortunately, sometimes one can’t do it the easier way. One has to do it the hard way.

But, this leads into something else I wrote on Quora.com: Do Bad Programmers Know hey write bad code? I only partly addressed the question, but to sum it up, I think the worst don’t, but the best programmers do, and sometimes intentionally.

The truth is, we probably should always be writing the best code we can. It should handle error trapping and handling. It should validate inputs. It should fail gracefully.

But often, we need a one-off script. Something that gets the job done here and now.

I recently did a 5-minute lightning round for my SQL User Group on the benefits of using PowerShell. I took to quick and dirty scripts I had written and rewrote them a bit for the presentation.  Afterwards, one of the attendees asked me a few questions about my stylistic choices in the code.  He was right in general, but I pointed out what my goal was. My goal was more to show what PowerShell could do than to actually show how to write good code in PowerShell.  That said, I probably should have written slightly better code, but this got the job done. It definitely didn’t need error handling and the like. It was good enough.

And ironically, this post is sort of like that. (I love it when I can get meta on my own posts). I have about a dozen drafts I have saved in WordPress. Most have just a title and a quick set of notes on what I think I should write about. This post was mostly written and just needed a bit more to flesh it out.  It was easier this way than trying to come up with a new topic for this week. Hope you enjoyed it. (and to keep things even easier, I’m going to let WordPress use a random photo for it!)

Snow Daze

I’ll be honest upfront. I have a love/hate relationship with snow. I love the sounds of a snowstorm. I don’t mean the wind if it’s there, I mean the quiet hush that can settle over the land, a muffled calm. It’s peaceful.

I love the starkness it can leave behind, everything white with a gray or black background. The lack of color is beautiful in its own way.

I love how it slows down life for a bit. You can’t go fast. You have to stop and take a measure of the moment.

And, I hate shoveling it. Sure, it’s a good workout, if you’re fit enough and not at a risk for a heart attack. But, it can be a pain. I drive a Subaru, so often I only shovel the end of my driveway where the snowplows leave their pile of detritus as they go by since once that freezes in place it can become an implacable wall. But this snowstorm, I did the entire driveway. It was deep enough I thought it prudent.  And it’s not just the weight of the snow that’s the issue. It’s moving it to a place that’s out of the way.

I’m a person who actually loves the seasons. I love how they mark the inexorable passage of time.

“One Christmas was so much like the other, in those years around the sea-town corner now, out of all sound except the distant speaking of the voices I sometimes hear a moment before sleep, that I can never remember whether it snowed for six days and six nights when I was twelve, or whether it snowed for twelve days and twelve nights when I was six.” – Dylan Thomas A Child’s Christmas in Wales

I’ve been alive for over a half-century now and at times it seems the years fly by faster. The snippet from Dylan Thomas reminds me so much of Falls Village CT, the town I grew up in. In some ways it never lost a certain Victorian charm.  We were part of the modern world, but separate enough that we were ensconced in a tiny bubble. In the winter, I’d lay awake on a snowy morning hoping to hear the snow plow go by.  We’d build forts in the snow and have epic snowball fights. We’d come in sopping wet and warm ourselves by the wood stove and drink hot chocolate and share epic tales amongst ourselves.

When it got a bit closer to Christmas we’d go to the nursery and buy a 14′ tall tree, because we could.  We’d decorate the first 8′ and then above that, the best we could.

It’s not Christmas yet here, but at GMS Headquarters it’s slowly arriving:

20191203_094342

Christmas Lights

And soon I’ll start listening to Christmas music.

And I’ll try not to think about the next round of shoveling I have to do.

I’d love to sit around all day sipping hot chocolate and doing nothing, but there’s work to do and eventually more snow to shovel.

And I can hear the occasional plow go by, getting the roads fully clear.

All is well here at GMS Headquarters.

 

Sleep, Perchance to Dream

I’ve skipped two weeks of blogging, which is unusual for me. The first time I’ll admit I was simply too tired after flying back from SQL Summit in Seattle. That, combined with catching up on work for my largest client meant I simply didn’t have time. And last week, well Monday night I was sure I was coming down with the flu and had a terrible night’s sleep and was in a brain fog all day Tuesday.  Now I have no idea if it was the flu (I tend to doubt it) but by Wednesday or Thursday I was feeling a lot better.

And, then last night, I barely slept either. So suffice to say, I’ve had sleep on my brain a lot lately.

I find sleep and dreaming to be fascinating aspects of evolution. When you stop to think about it, unless you’re an apex predator, sleeping would appear to be an evolutionary poor choice in many cases. Depending on the animal, it can spend anywhere from 2-3 hours asleep (perhaps broken up over the course of the day) to 20 hours.

And in fact, predators often tend to sleep more, which conserves energy, while prey tend to sleep less (so they can more easily flee said predators).

Some animals in fact are capable of unihemispheric sleep, i.e. only half of their brain appears to go to sleep. In fact dolphins in pods appear to sleep such that if they’re on the outside of the pod, the side of the brain that goes to sleep is opposite of the eye on the outside of the pod. They literally sleep with “one eye open” looking for danger.

Yet, despite the risks to prey, they still sleep. It seems pretty universal and something that as far as I know, all vertebrates do to some point. So it seems pretty necessary. And we’re learning at least in humans that chronic lack of sleep can lead to issues such as dementia later in life or even a shortened lifespan.  It seems the phrase, “you can sleep when you’re dead” tends to mean that your death may come even sooner if you chronically undersleep.

And when we sleep, we don’t just basically stop interacting with the outside world, we create a fantasy world inside our heads. I’m a person who can often remember his dreams and they tend to  be vivid and rich in detail. When dreams incorporate elements of places I’ve been or seen I’m not surprised. But then occasionally I will dream of places I know I’ve never been, houses I’ve never been in, landscapes I’ve never seen in such detail. It amazes me that my brain can, to use a computer term, render such rich detail in what appears to be real time.

In any event, right now, sleep is on my brain, but work calls.

Kids, get off my lawn!

Change can be hard. But sometimes it’s necessary. And a lot has happened this week.  First, I want to congratulate my fellow #SQLFamily member Cathrine Wihlemsen on one more orbit of the Sun. Apparently, in her honor Microsoft decided to release SQL Server 2019 on her birthday! I’ve been using SQL Server since the 4.21a days. Every version has had new features and required learning something new. As I said recently, it’s easy to fall into the trap of being an old dog and not learning new tricks. This is something we have to avoid. Being trapped in the past can be limiting.

Besides SQL Server 2019 dropping this week, I recently upgraded my phone. I had been using a Windows Phone for about 5 years now. I loved it. Especially when it first came out, it was top of the line and had a bright future. I eagerly downloaded apps and it became part of my life. But alas, we know how well Microsoft did in the Windows Phone market. But I doggedly held on, even as features were deprecated. I couldn’t use the Weather App. The Amtrak App went away. Eventually several features of Cortana stopped work as Microsoft stopped supporting them. Slowly my phone was becoming a brick. I kept debating do I upgrade to one more Windows Phone knowing it’s the end of the line, or what? I kept putting off the decision. After the mapping function failed me on my recent trip to the Hampton Roads User Group Meeting I decided it was time to finally time to replace it with an Android phone. Choosing from the plethora out there was not fun. It was very tempting to go with one of the top of the line models, but spending $1000 or so wasn’t really a fun idea.  I eventually ended up choosing a Samsung A50.

I’m mostly happy with it. Right now I’m struggling with what parts of it are “get off my lawn” because I don’t like change, and what parts are “what the hell is the UI doing now?”  Fortunately, my son has mentioned some of his dislike of certain UI functionalities, so I think not all of it is me simply being an old curmudgeon (are there young ones?) I will say what I’m most happy with is that Microsoft has a number of tools including the Windows Launcher and the Phone Companion, as well as the obvious apps like Outlook and other parts office.

A word about the Phone Companion. This alone has made the upgrade a win. One of the features is that when I’m working at home (I have not yet enabled it on my Surface Pro) is that things like text messages pop-up on my desktop screen. This actually makes life a LOT easier, since I can simply type a reply from a full-size keyboard or copy the numerous soft-tokens I get to log into various client sites without having to pick up my phone. It’s a small detail, but a wonderful one!

The Launcher helps me retain some of the features that I liked about my Windows Phone. Overall, it’s a win.

But the changes in my life aren’t complete. As I mentioned last week I’m at PASS Summit again this week in Seattle. But alas, this is the last year that PASS Summit will be in Seattle. Next year it will be held in Houston. Just as I’ve figured out where the cheapest and most convenient parking for me is, where some decent food places are, and I’m feeling, if not at home in Seattle, at least comfortable, next year is a big change. I won’t be able to stay with my college friends or do our annual Thai pot luck with a bunch of ROC Alumns.

But, I’ll get to explore another city. I’ve been to Houston only once, literally decades ago, to do SQL Server install at Exxon. The server was literally the only Intel computer in a room full of mainframe equipment. I suspect that has changed since then.  That was one of my early experiences installing SQL Server (4.21a for the record).

So, this old dog is still learning and looking forward to new experiences: plus ça change, plus c’est la même chose.

 

Small Victories

Ask most DBAs and they’ll probably tell you they’re not a huge fan of triggers.  They can be useful, but hard to debug.  Events last week reminded me of that. Fortunately a little debugging made a huge difference.

Let me set the scene, but unfortunately since this was work for a client, I can’t really use many screenshots. (or rather to do so would take far too long to sanitize them in the time I allocate to write my weekly blog posts.)

The gist is, my client is working on a process to take data from one system and insert it into their Salesforce system.  To do so, we’re using a 3rd party tool called Pentaho. It’s similar to SSIS in some ways, but based on Java.

Anyway, the process I was debugging was fairly simple. Take account information from the source and upsert it into Salesforce. If the account already existed in Salesforce, great, simply perform an update. If it’s new data, perform an insert.  At the end of the process Pentaho returns a record that contains the original account information and the Salesforce ID.

So far so good. Now, the original author of the system had setup a trigger so when these records are returned it can update the original source account record with the Salesforce ID if it didn’t exist previously. I should note that updating the accounts is just one of many possible transformations the entire process runs.

After working on the Pentaho ETL (extract, transform, load) for a bit and getting it stable, I decided to focus on performance. There appeared to be two main areas of slowness, the upsert to Salesforce and the handling of the returned records. Now, I had no insight into the Salesforce side of things, so I decided to focus on handling the returned records.

The problem of course was that Pentaho was sort of hiding what it was doing. I had to get some insight there. I knew it was doing an Insert into a master table of successful records and then a trigger to update the original account.

Now,  being a 21st Century DBA and taking into account Grant Fritchey’s blog post on Extended Events I had previously setup a Extended Events Session on this database. I had to tweak it a bit, but I got what I wanted in short order.

CREATE EVENT SESSION [Pentaho Trace SalesForceData] ON SERVER
ADD EVENT sqlserver.existing_connection(
    ACTION(sqlserver.session_id)
    WHERE ([sqlserver].[username]=N'TempPentaho')),
ADD EVENT sqlserver.login(SET collect_options_text=(1)
    ACTION(sqlserver.session_id)
    WHERE ([sqlserver].[username]=N'TempPentaho')),
ADD EVENT sqlserver.logout(
    ACTION(sqlserver.session_id)
    WHERE ([sqlserver].[username]=N'TempPentaho')),
ADD EVENT sqlserver.rpc_starting(
    ACTION(sqlserver.session_id)
    WHERE ([package0].[greater_than_uint64]([sqlserver].[database_id],(4)) AND [package0].[equal_boolean]([sqlserver].[is_system],(0)) AND [sqlserver].[username]=N'TempPentaho')),
ADD EVENT sqlserver.sql_batch_completed(
    ACTION(sqlserver.session_id)
    WHERE ([package0].[greater_than_uint64]([sqlserver].[database_id],(4)) AND [package0].[equal_boolean]([sqlserver].[is_system],(0)) AND [sqlserver].[username]=N'TempPentaho')),
ADD EVENT sqlserver.sql_batch_starting(
    ACTION(sqlserver.session_id)
    WHERE ([package0].[greater_than_uint64]([sqlserver].[database_id],(4)) AND [package0].[equal_boolean]([sqlserver].[is_system],(0)) AND [sqlserver].[username]=N'TempPentaho'))
ADD TARGET package0.ring_buffer(SET max_memory=(1024000))
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=ON,STARTUP_STATE=OFF)
GO

It’s not much, but it lets me watch incoming transactions.

I could then fire off the ETL in question and capture some live data. A typical returned result looked like

exec sp_execute 1,N'SourceData',N'GQF',N'Account',N'1962062',N'a6W4O00000064zbUAA','2019-10-11 13:07:22.8270000',N'neALaRggAlD/Y/T4ign0vOA==L',N'Upsert Success'

Now that’s not much, but I knew what the Insert statement looked like so I could build an insert statement wrapped with a begin tran/rollback around it so I could test the insert without actually changing my data.  I then tossed in some set statistics IO ON and enabled Include Actual Execution Plan so I could see what was happening.

“Wait, what’s this? What’s this 300K rows read? And why is it doing a clustered index scan on this table?”  This was a disconcerting. The field I was comparing was the clustered index, it should be a seek!

So I looked more closely at the trigger. There were two changes I ended up making.

       -- Link Accounts
       --MERGE INTO GQF_AccountMaster T
       --USING Inserted S
       --ON (CAST(T.ClientId AS VARCHAR(255)) = S.External_Id__c
       --AND S.Transformation in ('Account'))
       --WHEN MATCHED THEN UPDATE
       --SET T.SFID = S.Id
       --;
       
       if (select transformation from Inserted) ='Account'
       begin
              MERGE INTO GQF_AccountMaster T
              USING Inserted S
              ON T.ClientId  = S.External_Id__c
              WHEN MATCHED THEN UPDATE
              SET T.SFID = S.Id
       end

An astute DBA will notice that CAST in there.  Given the design, the Inserted table field External_Id__C is sort of a catch all for all sorts of various types of IDs and some in fact could be up to 255 characters. However, in the case of an Account it’s a varchar(10).

The original developer probably put the CAST in there since they didn’t want to blow up the Merge statement if it compared a transformation other than an Account. (From what I can tell, T-SQL does not guarantee short-circuit evaluation, if I’m wrong, please let me know and point me to definitive documentation.) However, the minute you cast that, you lose the ability to seek using the index, you have to use a scan.

So I rewrote the commented section into an IF to guarantee we were only dealing with Account transformations and then I stripped out the cast.

Then I reran and watched. My index scan of 300K rows was down to a seek of 3 rows. The trigger now performed in subsecond time. Not bad for an hour or so of work. That and some other improvements meant that now we could handle a few 1000 inserts and updates in the time it was previously taking to do 10 or so.  It’s one of those days where I like to think my client got their money’s worth out of me.

Slight note: Next week I will be at PASS Summit so not sure if/when I’ll be blogging. But follow me on Twitter @stridergdm.

Hampton Roads User Group Recap

I’ve talked about how I think it’s important to be part of the #sqlfamily community and how I enjoy talking and giving back. Last week was another example of this. Much earlier this year (it might have even been at Pass Summit last year) I convinced Monica Rathbun to do a quid pro quo. I’d speak at her user group in Virginia Beach if she’d come to upstate NY to speak at my user group. I’d seen Monica speak and knew she would be a great speaker for my group. Fortunately, despite seeing me speak, she apparently felt I’d be good enough for her group.  Seriously though it was a good deal.

My original plan had been to drive down Wednesday, address her group, stay at an AirBnB on the beach and then spend a few nights in the Washington DC area visiting with some friends.  Unfortunately, less than a week before I was ready to head down, my DC plans fell through. This radically changed my travel plans and I scrambled to make various plans to make the trip a practical one and one that wouldn’t break my budget. One of the unfortunate facts of being as consultant is that I don’t have an employer that can cover travel expenses. On the other hand, I often have a lot more flexibility in when and how I travel.

I ended up taking the train to Wilmington Delaware and getting a rental car from there. This allowed the most flexibility, was second in time to flying, and overall the least stressful. I love taking the train because I can sleep (which I did on the Albany to NYC segment) and get work done (which I did on the NYC-Wilmington segment, working on a future article for Redgate Simple-Talk and reviewing my talk) Unfortunately, due to a missed turn, some slow traffic due to the rain and then the rain in general, rather than showing up at 5:30 like I had hoped, I was in the door at 6:15 or so. This gave me time for a single chicken wing before I launched into my talk.

I had been monitoring the Meet-up page to see how many people were expected and at my last count it was 8. I was comfortable with that. I was hoping for more, but hey, I’ll take what I can get.  Imagine my surprise when I walked in and there were closer to 20 people there. Honestly, a great turnout! But, between running late, the usual hardware issues of getting my laptop and the monitor talking, and not being able to get one last run through of a 10 minute section of my talk, I’ve got to say I was a bit flustered.

I love to teach. But I would be lying if I said I don’t love it when I see or hear a student have what I call that “Aha moment!” This is that moment when you explain or demonstrate something and you can see the look in their eyes or the tone in their voice when something just clicks. It might be a small thing to you, but for them you’ve just rocked their world.

A number of years ago while teaching the Level 2 cave rescue class in Colorado, we were doing an instructor lead evolution. During these, the instructors take the lead and guide the students through the problem. It’s usually the first real new teaching experience of the week-long class, before that it’s mostly review. In this case I had a single student working with me and we were charged with setting up two lines to be used as a safety and for another purpose.  I told her to grab a single rope and a carabiner. She looked at me questioning because she knew we needed to have two lines rigged. I then showed her the tree I had selected and told her to basically double the rope, tie what’s known as a high strength tie-off using the middle of the rope, clip it in with the carabiner and toss both ends down. Then the aha moment, “wow, I’d never thought of that. That’s worth the price of the class right there.” I’ve got to say I was proud. My job was done, 2nd day of teaching. I could take the week off. Of course I didn’t.

This time around, I was talking about the Model Database and how most DBAs completely ignore it and overlook it. I was demonstrating how when you put objects in it or change various options in it (such as from Simple Logging to Full Logging) any new databases will pick up those objects or options (unless you override the options using a script.)  As I was bending over the keyboard to type the next demo I heard it, someone in the middle of the classroom suddenly said, “Woah…” and you could tell their world had just been expanded. That alone made the entire 36 hours (including travel time, sleeping etc) of the trip worth it. I knew someone had learned something. I live for those moments.

Don’t get me wrong, I enjoy getting paid as a consultant, but honestly, I speak on SQL Server topics and teach cave rescue for those aha moments, for knowing that I’ve just expanded someone else’s world a bit.

Oh that, and in this case, the free wings!

ehb5jp6w4ae8-at.jpg

Tasty wings at Hampton Roads SQL Server User Group

Just a reminder, I will be at the 2019 PASS Summit in Seattle and look forward to meeting with anyone who wants. My Twitter handle is @stridergdm and I often hang out with the folks at MinionWare (they’ve got a comfortable couch) and will be attending the Birds of the Feather luncheon (undecided where I’ll be sitting) and the Women in Technology Luncheon.

And I’m hoping for my nest article on PowerShell for Redgate’s Simple-Talk to be submitted before then!