Once upon a time I was in charge of a release for an important update to a large project. I had to coordinate the release with other teams in order for it to go smoothly.

Most projects I work on these days have release processes handled by a QA department wherein they press a button and things happen automagically. This project lacked a real release process. Here’s what a typical release looked like on this project:

git fetch origin

# manually review changes as a sanity check
git diff --name-only HEAD..origin/master
git diff HEAD..origin/master

# release if everything looks good
git merge origin/master

For the record, this is a terrile way to do releases, but it’s what we had.

For this big release I didn’t look at the full diffs because there were thousands upon thousands of modified files. I’d already spent days combing through the diffs. No need to do that again on production.

When the clock struck 12 and the office began to vibrate with the sound of the massive bells that we had set up just for this release, I hit return on git merge with great expectations and… there was a conflict. Surprise!

Now, in a normal situation with an unexpected conflict (i.e. there should be no modified files on production!), I’d perform a git status and git diff to see what was going on. If the modified files were important I’d do a git stash and then reapply the stashed changes after the merge.

But this was no normal situation. Other people were anxiously waiting for an update. We had a plan! My part of the release was supposed to take 30 seconds.

I tried to stick to the planned timeframe instead of doing the proper due diligence to see what was wrong. New information cannot slow me down! I’ve never seen this particular conflict on this server before! Surely I can plow onwards!

It’s like my brain threw an exception and I had no exception handling in place. All previous releases I’d performed on this server had been fine.

I did have the presence of mind to ask other people if the specific conflict mattered. It did not. So I proceeded with gleeful abandon and did a git stash without looking at what I was stashing. I thought I was stashing changes to one file and that file didn’t matter.

Unbeknownst to me, there were other files that had conflicts too.

I informed everyone that I had successfully deployed the code. Whew! Time for the next step, restarting the webserver!

This is basically what happened:

The server did not restart.

The site was down.

We all scrambled to figure out what had happened.

We eventually discovered that the server configuration was outdated and that it lived in the git repository (never do this). How it had become outdated, I had no idea. I hadn’t ever touched the file. It took an hour, but the server configuration was updated and the site came back online.

I didn’t realize what my mistake had been until the situation was resolved and my stress level dropped. At some point the webserver configuration had been updated on production. I didn’t know it was in the git repository. Had I looked at the output of git status or run a git diff I would have seen it and avoided the whole problem.

But I let the stress of the situation get to me. I placed too much importance on our timeframe and ignored the very obvious red flags staring me in the face.

I learned that when I am stressed I don’t naturally think through changes and new information clearly before acting. Instead, I panic and make things worse by trying to stick to an outdated or bad plan.

I haven’t completely solved this problem yet (being human and all), but I am getting better at it. I try to expect and plan for changes or complete failure. If we plan on a day at a water park and it pours rain, it’s okay to stay home and watch a movie. There’s no reason to be miserable in the rain all day.

It’s okay to slow down. It’s okay to rollback a release and evaluate what happened before proceeding. Arbitrary timeframes don’t matter. In fact, strict timeframes also don’t matter if you screw it up.

A good way to slow down in a high stress situtation is to eat cookies. The cookies will (hopefully) taste so good it’ll be hard to put them down, thus providing you with critical thinking time.

In fact, I’m so committed to this whole cookie thing that I’ve decided to pivot my entire career and become a Cookie Baking Release Engineer. Feel free to reach out if you feel this would be important for your business. I am also open to making donuts and ice cream.