Debugging – Powerhell (sic)

Generally I don’t plan to talk too much about specific programming problems. There’s too many blogs out there about how to solve particular problems; and most of them are generally good and even better, accurate.

I’m going to bore most of you, so if you want to jump to the summary, that’s fine. I won’t be insulted (heck I won’t even know!)

That said, last night I spent a lot of time debugging a particular PowerShell script I had written for a client.

I want to talk a bit about what was going on and how I approached the problem.

So, first the setup: It’s really rather simple

  1. Log into an SFTP site
  2. Download a zip file
  3. Expand the zip file
  4. Run an SSIS package to import the files in the zip package to a SQL Server

I had written the script several months ago and it’s been in production since then.  It’s worked great except for one detail. When it was put into production, we didn’t yet have the service account to run it under, so I set it up to run as a scheduled task using my account information. Not ideal, but workable for now.

Since then, we received the service account. So, I had made several accounts to move to the service account, but it wasn’t working.  Yesterday was the first time in awhile I had time to work on it so I started looking at it in more detail.

Permissions Perhaps?

So, the first problem I had was, I’d run the scheduled task and nothing would happen. Ok, that’s not entirely accurate. The script would start, but never run to completion. It just sort of sat there. And it certainly was NOT downloading the zip file.

My first thought was it might be a files permission or other security issue. So the first thing I did was write a really simply script.

get-date | out-file “D:\foo\timetest.text”

Then I setup a scheduled task and ran it manually. And sure enough, timetest.txt showed up exactly where it was supposed to.

So much for security in the OS.

out-file to the Rescue!

So, my next step is probably one of the oldest debug techniques ever: I put in statements in the code to write to a file as I hit various steps in the code.  Not quite as cool as a debugger, but hey, it’s simple and it works.

It was simple stuff like:

“Path $SftpPath Set” | out-file ftp_log.txt -append

BTW, if you’re not familiar with Powershell, one of the nice things is in the above statement, it’ll automatically insert the variable $SftpPath into the string it’s printing out.

This started to work like a charm. And then… jackpot!

“Setting Session using $sftpURL and Password: $password and credentials: $Credential” | out-file ftp_log.txt -append

When I ran it, I’d get a line in the log like:

Setting Session using secureftp.example.com and Password: System.Security.SecureString and Credentials: System.Management.Automation.PSCredentials

Not the most informative, but useful. And it’s nice to know that one can’t easily just print out the password!

But, when I ran it as the service I was getting something VERY different.

Setting Session using secureftp.example.com and Password:  and Credentials:

HUGE difference. What the heck was happening to the password and credentials?

That’s when the first lightbulb hit.

Secure credentials in PowerShell

Let’s take a slight side trip here.  We all know (right, of course you do) that we should never store passwords in cleartext in our scripts.  So I did some digging and realized that PowerShell actually has a nice way to handle this. You can pass a plaintext string to a cmdlet and write that out to a file.  Then when you need credentials, you read from the file and it gets parsed and handed to whatever needs it.

$password = get-content $LocalFilePath\cred.txt | ConvertTo-SecureString

Of course first you have to get the password INTO that cred.txt file.

That’s easy!

if (-not (test-path $LocalFilePath\cred.txt))
{
read-host -AsSecureString | ConvertFrom-SecureString | Out-File     $LocalFilePath\cred.txt
}

So, the first time the program is run, if the cred.txt file isn’t found, the user is prompted for it, they enter the password, it gets put into a secure string and written out. From then on, the cred.txt file is used and the password is nice and secure.

And I confirmed that the service account could actually see the cred.txt file.

So what was happening?

The Key!

It took me a few minutes and when I hit a problem like this I do what I often do, step away from the keyboard. I had to stop and think for a bit. Then it dawned on me. “How does Powershell know how to encrypt and decrypt the password in a secure fashion. I mean at SOME point it needs to have a clear-text version of to pass to the various cmdlets. It wouldn’t be very secure if just anyone could decrypt it.  That’s when it struck me! The encryption key (and decryption key) has to be based on the USER running the cmdlet!

Let me show some more code

$password = get-content $LocalFilePath\cred.txt | ConvertTo-SecureString
$Credential = New-Object System.Management.Automation.PSCredential (‘ftp_example’, $Password)

$sftpURL = ‘secureftp.example.com’

$session = New-SFTPSession -ComputerName $sftpURL -Credential $Credential

So first I’d get the password out of my cred.txt file and then create a Credential Object with a username (ftp_example) and the aforementioned password.

Then I’d try to open a session on the SFTP server using that credential.  And that’s exactly where it was hanging when I ran this under the service account. Now I was on to something. Obviously the $password and $Credential wasn’t getting set as we saw from my debug statements.  It wasn’t able to decrypt cred.txt!

Great, now I just need to have the service create its own cred.txt.  But, I can’t use the same technique where the first time the service runs it prompts the user for the password.  This bothers me. I still don’t have a perfect solution.

For now I fell back on:

if (-not (test-path $LocalFilePath\cred_$env:UserName.txt))
{
# after running once, remove password here!
“password_here” | ConvertTo-SecureString -AsPlainText -Force | ConvertFrom-SecureString | Out-File $LocalFilePath\cred_$env:UserName.txt
}

Note I changed the name of the cred.txt file to include the username, so I could have one when I ran it and another when the service account ran it. This makes testing far easier AND solves conflicts when debugging (i.e. “which credentials are currently being used).

So for now the documentation will be “after running the first time, remove the password”. I think more likely I’ll write one-time script that runs to setup a few things and then deletes itself. We’ll see.

Anyway, this worked much better. Now my debug lines were getting the password and all. BUT… things were STILL hanging.  Talk about frustrating. I finally tracked it down to the line

$session = New-SFTPSession -ComputerName $sftpURL -Credential $Credential

Caving Blind

As you know, I love caving. And for caving, a headlamp is a critical piece of equipment. Without it you have no idea where you’re going.  That’s how I felt here. Things were simply hanging and I was getting NO feedback.

I tried

$session = New-SFTPSession -ComputerName $sftpURL -Credential $Credential | out-file ftp_log.txt -append

But I still wasn’t getting anything. I tried a few other things, including setting parameters on how New-SFTPSession would handle errors and warnings. But still nothing. It was like caving blind. The cmdlet was being called, but I couldn’t see ANY errors.  Of course when I ran it as myself, it ran fine. But when I ran it as a service, it simply hung.

I was getting frustrated I couldn’t get any feedback. I figured once I had feedback, the rest would be simple. I was mostly right.

I needed my headlamp! I needed to see what was going on!

Finally it dawned on me, “wrap it in an exception”. So now the code became:

try
{
$session = New-SFTPSession -ComputerName $sftpURL -Credential $Credential
}
catch
{
$exception = $_.Exception.Message
“Error in New-SFTPSession – $exception – ” | out-file ftp_log.txt -append
}

Now THIS got me somewhere. I was getting a timeout message!  Ok, that confused me. I mean I knew I had the right username and password, and it never timed out when I ran it. Why was it timing out when the service ran it?

So again, I did what I do in cases like this. Got up, went to the fridge, got some semi-dark chocolate chips and munched on them and ran through my head, “why was the secure FTP session timing out in one case, but not the other?”  At times like this I feel like Jack Ryan in Hunt for Red October, “How do you get a crew to want to get off a nuclear sub…”

Eureka!

DOH, it’s SECURE FTP. The first time it runs it needs to do a key exchange!  I need to accept the key!  At first I panicked though. How was I supposed to get the service to accept the key?  Fortunately it turned out to be trivial. There’s actually a parameter -Acceptkey. I added that… and everything ran perfectly.

Except the SSIS package

That’s a separate issue and again probably related to security and that’s my problem to debug today. But, I had made progress!

Summary

Now, quite honestly, the above is probably more detail than most care to read. But let me highlight a couple of things I think that are important when debugging or trouble shooting in general.

First, simplify the problem and eliminate the obvious. This is what I did with my first script. I wanted to make sure it wasn’t something simple and obvious. Once I knew I could write to a file using the service account that ruled out a whole line of questions. So I could move on to the next step.

Often I use an example from Sesame Street of “one of these things is not like another.”  In this case, I had to keep figuring out what was different between when I ran it and when the service account ran things. I knew up front that anything requiring keyboard input would be a problem. But, I thought I had that covered. Of course starting to compare and contrast the results of decrypting the cred.txt file showed that there was a problem there. And the issue with the initial acceptance of the SFTP key was another problem.

So, gather information and compare what works to what doesn’t work. Change one thing at a time until you narrow down the problem.

The other issue is being able to get accurate and useful information. Using debugging statements if often old school, and I know some developers look down on them, but often the quick and dirty works. So I had no problem using them. BUT, there are still cases where they won’t work. If for example a cmdlet hangs or doesn’t give output to standard output, they won’t work. Catching the exception definitely solved this.

The biggest problems I really ran into here is I’m still a beginner with PowerShell. I’m loving it, but I often still have to lookup things. But the basic troubleshooting skills are ones I’ve developed and honed over the years.

And quite seriously, sometimes don’t be afraid to walk away and do something else for a bit. It almost always brings a fresh perspective. I’ve been known to take a shower to think about a problem. Working from home that is easier for me than it might be for someone in an office. But even then, go take a walk. Or do something.  Approach the problem with a fresh mind. All too often we start down the wrong path and just keep blindly going without turning around and reevaluating our position.

When we teach crack and crevice rescue in cave rescue we tell our students, “If you’re not making progress after 15 minutes, let someone else try. The fresh approach will generally help.”

Looking back on this, the problems seem pretty obvious. But it took a bit of work to get there. And honestly, I love the process of troubleshooting and/or debugging. It’s like being a detective, compiling one clue after another until you get the full picture.

So now on to comment some of the code and then figure out why the SSIS package isn’t executing now (I have ideas, we’ll see where I get to!)

Crane Operators

Talking online with friends the other day, someone brought up that crane operators in NYC can make $400-$500K a year. Yes, a year. I figured I’d confirm that before writing this post and it appears to be accurate.

At first glance one may think this is outrageous, or perhaps they chose the wrong field. I mean I enjoy being a DBA and a disaster geek, but I can’t say I’ve ever made $400K in one year!  And for what, I mean you lift things up and them down. Right?

Let me come back to that.

So, last night, I got paid quite a tidy bundle (but not nearly that much) for literally logging into a client computer, opening up VisualCron and clicking on a task and saying, “disable task”. On one hand, it seemed ridiculous;  not just because of what they were paying me, but because this process was the result of several meetings, more than one email and a review process.  All to say, “stop copying this file.”

But, this file was part of a key backup process for a core part of the client’s business. I had initially setup an entire process to ensure that a backup was being copied from an AIX server in one datacenter to a local NAS and then to the remote datacenter.  It is a bit more complex than it sounds.  But it worked. And the loss of a timely backup would impact their ability to recover by hours if not days. This could potentially cost them 100s of thousands of dollars if not into the millions.

So the meetings and phonecalls and emails weren’t just “which button should Greg click” but covered questions like, “do we have the backups we think we have?” “Are they getting to the right place(s)?” “Are they getting there in a timely fashion?”  And even, “when we uncheck this, we need to make sure the process for the day is complete and we don’t break it.”

So, me unchecking that button after hours, as much as it cost the company was really the end of a complex chain of events designed to make sure that they didn’t risk losing a LOT of money if things went wrong. Call it an insurance payment if you will.

Those crane operators in NYC? They’re not simply lifting up a beam here and there and randomly placing it someplace. They’re maneuvering complex systems in tight spaces with heavy loads where sudden gusts can set things swaying or spinning and a single mistake can do $1000s in damage or even kill people.

It’s not such much what they’re being paid to do, as much as how much they are being paid to avoid the cost of a mistake. I wasn’t paid just to unclick a button. I was paid (as were the others in the meetings) to make sure it was the right button and at the right time and that it wouldn’t cost even more.

Sometimes we’re not paid for what we do, as much as we’re paid for what we’re not doing.

 

This is secure, right?

WP_20180114_001So, over the weekend, while I was waiting for my wife’s hockey game to begin I was exploring the facility. I saw an upper level and I was curious if it was open. But alas no, it was locked. And I mean locked. Above is a photo of the chain and lock used to keep people out.

There’s a saying that locks only keep out honest people. That’s not entirely true, but there’s some truth to that. Security, especially in my field of IT and databases is extremely important. I am often working with data that contains private information about people. I have to take steps to keep it safe. So, as I mentioned in Too Secure, I don’t have a problem with security per se. There I talked about security that prevented me from doing my job.

Here we have the opposite. Making something that look secure, but really isn’t. I mean this has a heavy chain and a lock. I’m not sure who did this or why, but I’m sure they felt the lock was important. It’s not a cheap one either.

 

This is a case where probably a simple sign and string, “Balcony closed” would have sufficed.  The only purpose the lock really had was to make sure the chain wasn’t stolen. The purpose of the chain appeared to be to give something to attach the lock to.

Now, sure, someone could have removed a sign and string and said, “oh we never saw it” but again, what’s the ultimate risk if someone got upstairs? It was simply an observation balcony. They weren’t really securing much.

But I’m sure someone felt good about their security.

So, when you’re making something secure, stop and make sure you’re spending your time wisely. Sometimes not everything needs the most secure system available. Sometimes a “keep out sign” might be enough and easier.

Time, what is it?

Time is a funny thing. We all experience the passage of it, but, fundamentally it’s arbitrary.  In the SI system of units, the second is defined as “the duration of 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the caesium 133 atom”,

Sure. That’s helpful. To someone. Not to me.

It’s now 2018 AD. Or is that 2018 CE? Or is it 4714, or 1438, or perhaps 5778 or perhaps it’s 2010. Or perhaps it’s year 1?

We can’t agree on what year it is. But we can certainly at least agree on some semblance of the date, correct? Not really. If I’m using Julian dates, my day is going to be 12 hours off of yours.

But, since I’m writing this in the Northeastern US, let’s agree we’ll work with the Common Era Calendar and that it’s January 9th, 2018.  That should work.  Everything should be simple right? For many of people, it probably is. For us programmers, it isn’t.

Let’s start with a simple sentence, “Run the job at midnight on Saturday.” Can we agree on when to run it? Most of us would probably run it just as Saturday is beginning. But some would argue that it’s just as valid run it at the end of Saturday.  I have in fact seen both arguments. This is where our common use of the language can fall apart. Fortunately though, we can remedy the issue. The military for example, for whom precision timing can be critical would say something like, “Run the job at 00:00 on Saturday.”  Now at least we’re in agreement.

But even then, we love to muck things up. Let me start with being a pedant for a second and point out that there is technically no such thing as 12 PM or 12 AM.  “What?”, you might say? “Of course there is!” Nope, sorry. Going back to the Latin PM and AM mean Post-Meridiem and Ante-Meridiem, i.e. after and before the middle of the day. So technically, what we call 12 Noon can’t be either (since it’s neither before nor after the time of the middle of the day, it IS the middle of the day). And equally, one can argue that 12 Midnight is neither.  And yet, you hear folks use the terms anyway.

Almost always, but not always though 12PM is associated with Noon and 12AM with Midnight. This follows because 12:00:01 IS after the middle of the day. So it makes sense to call the second before that also PM. Yet, that said, I have seen multiple times signs claiming that a place is open for dinner from say 7:00PM-12:00PM and or open “All Day from 7:00 AM-12:00PM” and they’re clearly open after lunch and even after dinner. (And by the way, does it strike you as a strange that we count 12AM, 1AM, 2AM etc to 11AM.. and start over at 12PM, 1PM… i.e. the 12 is considered part of the next set of hours even though we immediately move back to 1 after it? It’s one reason I really prefer a 24 hour clock.)

So I get pedantic about this when folks say 12PM or 12AM and ask them to clarify. This is especially important when it comes to scheduling tasks in say SQL Server.

BUT, we have yet another problem. I’m in the Eastern Time Zone of the US. You might be in the Pacific Time Zone of the US.  I want to hold a meeting at 3:00 PM.  Is that my time or your time?  Most modern scheduling software will correctly put this into my calendar as a 3:00 PM meeting and into yours as a 12 Noon meeting.

Yet that’s still not enough. I was recently reminded of this a few months ago as I started to schedule my PASS Summit into my calendar. I put the events in while I was in the Eastern Timezone, forgetting that when I got to Seattle, my calendar would “nicely” shift things for me 3 hours.  This meant rather than say thinking I wanted to attend the 9:00 AM Summit, I wanted to attend a 6:00 AM Summit.  Fortunately again, the software I discovered DOES let me specify what timezone the appointment actually is in when I set it up.

And one more aside. If we schedule a meeting at 3:00 PM (and we both agree on what that means) what happens if I say “Oh, I have a conflict, can we move that back an hour?” We we moving it backwards in time to 2:00 PM, or backwards in the sense of “further away” and mean 4:00 PM?

So, we can agree that hours are tough. But it can get worse.  Here’s a question for you. When was George Washington born? My father was always proud that he shared George Washington’s Birthday. But did he?  Was it February 22nd 1732? What if I told you he was born February 11th 1731? Both are technically correct (which is the best kind of correct.)

We all know about leap years. We have them because it doesn’t take exactly 365 days to orbit the sun. It takes closer to 365.25 days to orbit the sun. But even THAT isn’t really accurate enough. When Julius Caesar reformed the Roman Calendar, he added the concept of leap years. He know that without them, the calendar would slowly get out of synch with the seasons. Before you knew it, July would start becoming the middle of winter and who would want that! (Ok, technically it would take a few hundred years, but you get the point).

The problem was, scheduling Leap Years still wasn’t enough. By 1582, despite leap years, the calendar had “slipped” by 10 days.  So Pope Gregory instituted the current calendar most of the world uses which added the more complex rule of leap years every 4 years, unless the year is divisible by 100 which are not leap years, except in the cases of when it’s divisible by 400 when it is a leap year.

Now, Pope Gregory had a lot of influence over parts of Europe, so predominately Catholic countries adopted the changes almost immediately, in 1582.  But understandably, the more Protestant countries were a bit slower to adopt. The Great Britain and the Colonies didn’t adopt it until 1752, after George Washington’s birth. So, he was born on either day depending on which calendar you’re using. And by then things had slipped 11 days. (The change in years is a related, but different issue that had to do with what was considered the first day of the year. It wasn’t always January 1st you know.)

Now, when Y2K rolled around, I have to admit, I started to wonder, and still do, what would have happened if we had had computers in 1899. Would they have gotten the lack of a leap year correct? My guess is, “probably not”. And in fact, ironically enough, I recall at least one software package (I can’t recall the name) that is well prepared for 2100 and 2200 and 2300, but was NOT prepared for the year 2000 and in fact skipped the leap day that year. (Oh, by the way to really blow your mind, depending on the calendaring system, the leap day is actually February 24th. Or perhaps a more accurate way of saying it is that the leap day is inserted after the 53rd day of the year.)

In conclusion let’s just say time been on my mind a lot lately, partly because I’ve orbited the Sun just over 50 times now and because it’s a new Year. And as a developer, at times handling dates can be far tougher than it looks at first pass, you have to deal with midnight boundaries, timezones, leap years (or the lack of them) and even wholesale shifts in calendars (and this is ignoring completely different calendars such as the numerous lunar based ones that may or may not have entire leap months!) Oh and completely left out any mention of leap seconds! Yes, you can have December 31st 23:59:60 as a valid time.

But when you think about it, other than the fact that time keeps slippin’ (slippin’, slippin’) into the future, all measurement is really arbitrary.  We pick a 0 point and go with it. We just have to agree at some point (in time, yes, the irony is not lost on me) what that 0 point is and how we’ll measure from there, and we’re all good.

Or are we? I’ll leave you with this one final thought. Far into the future, possibly trillions of years, when the Universe has expanded so much that the distance between particles is far enough that none can interact, essentially nothing will be changing. Can you even have time if you have no clock against which to measure it? Will time eventually run out?

I could go on and on about time, but, I seem to have run out of it.

 

Starting off the New Year… wrong

Ok, I had plans for a great blog post on time and how we measure it and how it really is arbitrary. But… that will have to wait. (I suppose that brings its own irony about time and waiting and all that. Hmm, now I may need a post about that!)

But, circumstances suggested a different post.

This one started with an email from my boss last year (who will remain nameless for his sake. 🙂  “Greg, I hacked together this web page so my team-members can track their hours. I need you to make a few tweaks to it. And don’t worry, we’ll replace it by the end of the year.”  Yes, that sound you hear is your head hitting the desk like mine, because you KNOW what is coming next. Here we are in January of 2018 and guess what, the web page is still in place.

And of course, it’s not just HIS (now former, he’s moved on) team that is using it. Apparently his boss liked what it could do, so his boss (my boss^2) declared that ALL his teams would use it.  So instead of just 4-5 people using it, there’s close to 30-40 people using it.

So, remember the first rule of development, “Oh don’t worry about this, we’ll replace it soon” never happens.  His original code made a lot of assumptions (which at the time I suppose seemed reasonable). For example, since many schedules are formed around work-weeks, he was originally storing hours by the week (and then day of the week) they were worked in. For example, if someone worked for 6 hours on January 5th of last year, the table stored that as “WEEK 1, day 5”.

Now, before anyone completely thinks this is nuts, do keep in mind, many places, including my last job at GE did everything by Week of the Year.  There’s some advantages to this (that I won’t go into here).

For the most part, the code worked and I didn’t care. But, at one point I was asked to do a report based not on Work Weeks, but on an actual calendar month. So, I had to figure out for example that February 1st occurred during Work Week 5, on the 4th day. And go from there.

So, I hacked that together.  Then where was a request for another report and another. Pretty soon it was getting to be a pain to keep track of all this. And I realized another problem. My boss had gotten lucky in 2017. January 1st occurred on a Sunday, which meant that Work Week 1, Day 1 was a natural starting point.

I started to worry about what would happen when 2018 rolled around. His code would probably break.  So I finally took the plunge and started to refactor the code.  I also added new features as they were requested.  Things were going great.

Until yesterday.  So now we get to another good rule of development: “Use Version Control” So simple. Even for a one person shop like me it’s a good idea. I had put it off for awhile, but finally downloaded the git plug-in for Visual Studio and started to use it.

So yesterday, a user reported that they couldn’t enter hours for last week. I pulled up the app and realized that yes, in fact it was coded in such a way you couldn’t go back into a previous year to enter hours. No problem, I can fix that!

Well, let me add a rule I had sort of missed; “Understand HOW to use Version Control”. I won’t go into details, but let’s just say, I wasn’t committing like I should have been or thought I was. So, in an attempt to get a clean base and all, I merged things… “poorly“. I had the branches I wanted, but had not properly committed stuff to them.

The work of the last few weeks was GONE. I know, you’re saying, “just go to your backups Greg, because of course a person who writes about DR has proper backups, right?”  Yeah, right! Let’s not go there!

Anyway, I spent 4 hours last night and 2 this morning recreating the code. Fortunately, it dawned on me, being .NET code, it was really just CLR and that perhaps with a good decompiler, I could get most of the code back without too much effort. I want to give props to RedGate’s .NET Reflector. Of the two decompilers I tried, it was clearly the better one. While I lost my variable names and the decompiled code isn’t quite what I’d call human written, it was close enough I could in short order recreate hours and hours of work I had done.

In the meantime, I also talked to some of my programming buddies on an RPI chat server (ask me about it sometime) about git and better procedures.

And here’s where I realized my fundamental mistake. It wasn’t just misunderstanding how Visual Studio was interacting with git and the branches I was creating, it was that somewhere in the back of my head I had decided that commits should be used only when major steps were completed. It was almost like I was afraid I’d run out; or perhaps complicate the history with too many of them.  But, you know what? Commits aren’t scarce resources. And I’m the only one reading the history. I don’t really care if I’ve got 10 commits or 100. The more the better in many ways. Had I been making commits much more frequently, I’d have saved myself a lot of work. (And I can discuss having a remote store at some point, etc.)

So really, the point of this hastily, sleep-deprived written blog isn’t so much to talk about dates and apps that never get replaced, but about a much deeper problem I was having. It wasn’t just failing to fully understand git commands, it was failing to understand how I should be using git in the first place.

So, to riff on an old phrase, “Commit Early, Commit often”.

Oh and I did hack together a way for that user to enter hours for last year. It’s good enough. I’m sure I won’t need it a year from now. The code will be all replaced or refactored by then I’m sure!

 

Maps

This is actually about a bit more than simply maps. It’s really about what a map is. It’s an abstraction. This may sound obvious, but it’s an important concept.

“I have a map of the United States… Actual size. It says, ‘Scale: 1 mile = 1 mile.’ I spent last summer folding it. I hardly ever unroll it. People ask me where I live, and I say, ‘E6.” – Stephen Wright

Forget the impracticality of such a map, it makes a fine joke, it’s still just a map. We don’t live on maps.  Piaget and others have done a lot of research into the development of the minds of young children and it’s fascinating. At first, I think one can argue they live in a very solipsistic world. Nothing exists except what they can see or touch. Soon though concepts of object permanence come into being. But they’re still not quite at the concept of true abstraction. At very young ages, if you hold up a picture of a room to a small enough child, they’ll try to grab stuff of the picture. And why not? They can grab things in the physical world.  Soon they start to realize the picture is just that, not the actual room, but a flattened portable version of it.

Around age 5 (I’m going on memory here) we start to ask children to draw maps. It might be a map of their room, or a map of how to get to school. They’re crude and often, to our adult minds wildly inaccurate, but, they’re starting to achieve a truer concept of abstraction.  Their maps aren’t even miniaturized pictures of the real world, they’re completely new, abstract designs of the world.

Eventually, around 8th grade or so, they’re probably introduced to Algebra. Up until then much of their math might have been things like 8 * ___ = 56. They’ll dutifully fill in the 7. It’s a blank, but it’s not really abstract. Then they start to solve problems like “56/x = 8, solve for x”.  Again they’ll write down 7, but now they’ve learned that a letter can represent a number and be used that way later on.

What’s interesting about the above is, many people never stop to realize that * or / or + or many other mathematical symbols are really just abstractions. Yes, numbers are in a sense an abstraction. There’s nothing inherent about the symbol 5 that means it MUST represent five of an object. It’s just a convenience (and it’s shorter than writing ///// hash marks). But the mathematical symbols are even more of an abstraction. If I wrote 8♒︎__ = 56, most of us could probably figure out what ♒︎ represents there.

Anyway, that aside, why am I writing about maps and ultimately abstraction? Because, it is so vitally important to my job and perhaps the jobs of most of my readers and it’s been on my mind lately.

I’ve been working on an SSIS package for a client. The original spec was “Import these 20 or so (out of 240) CSV files out of this ZIP file.”  I thought about it and after teaching myself a bit more about SSIS (I don’t usually do much in it) I realized I could create a container that did most of the work. And I could assign variables to the container. The variables include things like what database connection to use and where the files might be located. And, most importantly, a variable for the NAME of the file/table (by using a table with the same basename as the file, I can use a single variable and line stuff up nicely.) And part of what I had to (re)-learn was the scope of variables within SSIS packages and how to expose the ones I wanted as parameters.

Now, honestly, it probably took me more time to set up this level of abstraction to make it workable than if, for the 20 or so tables I had hard-coded everything.

BUT, I’ve got enough experience to guess at what was coming next; and I’m going to guess you know what’s coming next.

“Hey, this is great, but you know what, we think we want all 240 files in that ZIP file loaded. Can yo do that?”

“Sure!”  And at this point, the problem isn’t the amount of work, it’s the tediousness of it; “create table schema, create a new file connector, copy the SSIS container, update the variable for the table, go in and reconnect the file connect to the new table, test”.  It’s a LOT less work than if I had to hardcode everything by hand.  The container name even automatically is updated based on the table variable. This actually has an unintended, positive side-effect. If for some reason I start to duplicate a container (say I forgot I had already imported table Customer_Val, the container name won’t update using the variable since you can’t have duplicate container names. I caught this very problem the other day. So this was an unintended, but very useful additional side effect!)

And of course, once I move the final code to their UAT environment and then the Prod environment, it’s simply a matter of updating 3-4 variables in the deploy (for the database server, file location and where to set emails) and I’m done.

By taking the time to abstract out as much as I could, I’ve made my life easier. Up front it was definitely a bit more work, but in the long run, has saved me a lot of effort.

Nothing revolutionary about the concept to us developers. BUT, stop and try to think of an environment where maybe variables or such layers of abstraction don’t exist. Such environments exist and there’s some valid reasons for them, but ultimately, being able to abstract stuff can make our lives MUCH easier and our code that much more powerful.

And it appears that abstraction at this level is perhaps the ONE real intellectual advantage we have over other intelligence on this planet. And I’m not even so sure about that!

So I’ll end in pointing out that even the written word is really just a level of abstraction. So this column is about itself. Or something like that.

 

Star Wars: The Last Jedi

Ok, fair warning there may be spoilers ahead.  So unless you’ve seen the movie, or don’t care, postpone reading this particular post for awhile.

So let me start by saying, I loved it. I had some issues with it, but overall I loved it.

But, this post isn’t a movie review per se. It’s more about how a story might be crafted.

I have a daughter, 14 now, who was 12 when The Force Awakens came out. I can’t recall when she became a Star Wars fan, but she’s a huge fan and I love that about her.  I introduced her to the original trilogy and, only because the movies are about the good and the bad, the prequels. I still recall while watching The Phantom Menace she turned to me and said, “Dad, this makes no sense.”  She was right. But that didn’t stop her from watching all of them, The Clone Wars and pretty much anything else she could get her hands on that involved Star Wars.

So, 2 years ago, she told me for my birthday she wanted to take me to The Force Awakens. We went after school a day or two after it opened. The theater we went to was nearly empty, and honestly, not a great one. (I’ve come to really enjoy stadium seating, if only so it reduces the chances of someone’s head in front of me).

The theater darkened, the previews passed and that iconic chord rang through my ears and the scroll started; and I was looking at my daughter’s face. Suddenly, I was transported back to 1977 and I was 9 and I was watching Star Wars (it had no subtitle back then, let alone episode number) for the first time, all over again. I realized that at that point it didn’t matter if the movie was good or not (it was) I was being given a gift she didn’t realize she was giving. I was watching Star Wars through the eyes of a child.

That said, I did watch the movie. But I did turn to look at her one more time; the scene where Han is confronting is son, telling him to do what he knows he must do.  I knew what was coming. I also knew my daughter didn’t. The look of shock on her face as Kylo committed patricide was clear and apparent. She was shocked and upset.

We left the theater and she was clearly upset. “How could Han die! He’s a major character.” We stopped for dinner at Moe’s Southwest Grill and we talked. I explained how the scene almost certainly HAD to happen; that the movie was about Poe, Rey, Finn; that in a sense, the past had to fade to the past or even be killed off so that the story could advance. I also explained how Kylo’s character had to develop and that this was one very definite way to do so.  She wasn’t happy then.  A day later though she understood my point.

I think this was the point where she went beyond simply watching the story and trying to understand WHY a story is crafted the way it is. Sometimes it’s obvious, sometimes, not so obvious. Or a story may go in a direction the viewer may not like.

<SPOILER COMING>

Last warning!

So, again, this year she took me to see a Star Wars movie for my birthday. It’s a tradition I definitely enjoy. On the way to The Last Jedi, we talked about what Luke would do when she handed him his original light saber. I jokingly, but only 1/2, said, “eh, he’s going to throw it over his shoulder.”  So of course when the scene happened, I watched her face. It was worth it.  I say only 1/2 jokingly because again, to advance the plot and to make it more successful, I felt the movie had to break with the past. Him simply becoming a Jedi Master again much like Obi-Wan would have just made that part of the movie a rehash. Instead, we explored new ground. What happens when the Master doesn’t want to be? How does the apprentice react? How do WE as an audience react?

Luke asks Rey several times, “Why are YOU here.” She continues to respond with “Leia sent me”. Luke doesn’t accept that answer and neither do we.  We even know Rey knows it’s not the honest answer.  She finally gives the honest answer. But it’s not until much later in the movie she finally learns what is important; something far more important than anything a Jedi Master can teacher her, and it’s the one lesson that Kylo learns in the negative.  And as any well crafted movie, they learn their opposite lessons from both the exact same place and from very different places. But I won’t say more.

BUT, as I do try to tie these blog posts into something relevant to my overarching goal of this blog:

Ask yourself; “Why am I here?” “Why am I doing this job?” I once interviewed a candidate for a position. His answer was, “For the money.” While important, it wasn’t the answer I needed to hear. There’s always another, higher paying job down the street.  If you’re in it for only the money, you’ll never be content because you’ll always want a bit more. (that said, I don’t suggest working for less than you’re worth either.)

And when facing a problem. Don’t simply ask, “what happened” “Oh, Kylo killed Han”. No, ask yourself, “Why did it happen?” or “Why did it have to happen THAT particular way.” “Kylo killed Han to advance the plot and his character development.”

Ask the question behind the question. Get to the truth.

And every once in awhile, take the time to see the world again through the eyes of youth.