Weeknotes 2022 W39: Dance

September 26​–​October 2, 2022
1200 words

My friend Steve stayed over for a couple of days this week, which was good fun. The days ended up being longer than anticipated, with a bit too little sleep as a result. I really need my eight hours a night, it turns out. (The lack of sleep is a bit of a theme.)

Despite having used Git for many years now, I bumped into an entirely new problem with it earlier this week.

It turns out that if you have a branch with a certain name (say fix), then you cannot have any other branch starting with that name followed by a slash. Take a look:

% git checkout -b fix
% git checkout -b fix/bug-123

The error message that you get it as follows:

fatal: cannot lock ref 'refs/heads/fix/bug-123': 'refs/heads/fix' exists; cannot create 'refs/heads/fix/bug-123'

That is a terrible error message. I had to browse Stack Overflow for quite a while until I figured out what was going on.

At work, the branch naming convention is <type>/<description>, with type being fix or feature or so, and the description being the usual branch name. This is fine — at least until someone creates a branch named fix, preventing anyone else from creating any bug-fix branches.

I’ve been thinking about the use of approval gates in software engineering again, following a conversation I had with Tom Stuart. Behold my mostly unrefined thoughts below.

At most companies, I’ve observed that the process for getting a change into production consists of the following approval gates (all mandatory):

  1. Code review
  2. Manual acceptance test
  3. Manual regression test

There are plenty of variations on this, but in essence this is what most engineering teams use. The idea is that by having lots of reviews, the chance of introducing bugs in production gets lowered.

This isn’t quite true, though. Code reviews are not an effective way of finding bugs. Acceptance testing and regression testing can help, but if you want to do that for every deploy, you’ll end up with a time-to-production that is days or even weeks long. Also, this will soak up so much time that could have been used in other ways.

En­gi­neer­ing teams are so focused on preventing defects in production that they mostly ignore recovering from defects. The latter is important too, because the impact of defects is a factor of the following:

By putting a focus only on decreasing the number of defects, the time to recover goes up (because the time it takes to get something in production is so high), and the severity of the defects goes up as well (because deployments are done in large batches).

When the process for getting things in production has so many approval gates, deployments are done in large batches, presumably to save time. Doing the process for every single changeset could easily take up an entire engineer’s worth of time.

I’ve witnessed, first-hand, defects that take days (instead of hours or minutes) to resolve, because the process of getting stuff to production is so cumbersome.

In my mind, it’s better to have five defects that each take an hour to resolve, rather than one defect that takes days to resolve. Not only is the total impact of the defect lower, it frees up time for engineers to work on other, more useful things.

Increasing the time to production by adding more approval gates is counterproductive, as the time to recover goes up even more and the severity of the defects increases. When an engineering team decides that each changeset requires two mandatory code reviews, for example, I know things are going from bad to worse.

Additionally, being able to ship code fast encourages keeping the codebase well-factored, because refactorings will also be shipped quickly and therefore not cause issues like merge conflicts down the line. In my experience, if it is cumbersome to get changes in production, engineers will tend not to bother with refactoring the codebase. A well-factored codebase is a desirably healthy codebase, where test coverage is high and changes are unlikely to create new defects.

In one corner of my apartment, I have a fruit fly infestation. In the opposite corner of my apartment, I have a spider. The spider is not helping. Yummies are over on the other side, my friend.

Entertainment:

Links:

You can reply to this weeknotes entry by email. I’d love to hear your thoughts!
If you like what I write, stick your email address below and subscribe. I send out my weeknotes every Sunday morning. Alternatively, you can subscribe to the web feed.