Once upon a time I was in charge of a release for an important update to a large project.
I had to coordinate the release with other teams in order for it to go smoothly.
Most projects I work on these days have release processes handled by a QA department
wherein they press a button and things happen automagically. This project lacked
a real release process. Here’s what a typical release looked like on this project:
git fetch origin
# manually review changes as a sanity check
git diff --name-only HEAD..origin/master
git diff HEAD..origin/master
# release if everything looks good
git merge origin/master
For the record, this is a terrile way to do releases, but it’s what we had.
For this big release I didn’t look at the full diffs because there
were thousands upon thousands of modified files. I’d already spent days combing
through the diffs. No need to do that again on production.
When the clock struck 12 and the office began to vibrate with the sound of the
massive bells that we had set up just for this release, I hit return on
with great expectations and… there was a conflict. Surprise!
Now, in a normal situation with an unexpected conflict (i.e. there should be no
modified files on production!), I’d perform a
git status and
git diff to
see what was going on. If the modified files were important I’d do a
and then reapply the stashed changes after the merge.
But this was no normal situation. Other people were anxiously waiting for an
update. We had a plan! My part of the release was supposed to take 30 seconds.
I tried to stick to the planned timeframe instead of doing the proper due
diligence to see what was wrong. New information cannot slow me down! I’ve
never seen this particular conflict on this server before! Surely I can plow onwards!
It’s like my brain threw an exception and I had no exception handling in place.
All previous releases I’d performed on this server had been fine.
I did have the presence of mind to ask other people if the specific
conflict mattered. It did not. So I proceeded with gleeful abandon and did a
git stash without looking at what I was stashing. I thought I was stashing
changes to one file and that file didn’t matter.
Unbeknownst to me, there were other files that had conflicts too.
I informed everyone that I had successfully deployed the code. Whew! Time for
the next step, restarting the webserver!
This is basically what happened:
The server did not restart.
The site was down.
We all scrambled to figure out what had happened.
We eventually discovered that the server configuration was outdated and that it
lived in the git repository (never do this). How it had become outdated, I had
no idea. I hadn’t ever touched the file. It took an hour, but the server
configuration was updated and the site came back online.
I didn’t realize what my mistake had been until the situation was resolved and my
stress level dropped. At some point the webserver configuration had been updated
on production. I didn’t know it was in the git repository. Had I looked at the
git status or run a
git diff I would have seen it and avoided the
But I let the stress of the situation get to me. I placed too much importance on
our timeframe and ignored the very obvious red flags staring me in the face.
I learned that when I am stressed I don’t naturally think through changes and new information
clearly before acting. Instead, I panic and make things worse by trying to stick
to an outdated or bad plan.
I haven’t completely solved this problem yet (being human and all), but I am
getting better at it. I try to expect and plan for changes or complete failure.
If we plan on a day at a water park and it pours rain, it’s okay to stay home
and watch a movie. There’s no reason to be miserable in the rain all day.
It’s okay to slow down. It’s okay to rollback a release and evaluate what
happened before proceeding. Arbitrary timeframes don’t matter. In fact, strict
timeframes also don’t matter if you screw it up.
A good way to slow down in a high stress situtation is to eat cookies. The cookies
will (hopefully) taste so good it’ll be hard to put them down, thus providing
you with critical thinking time.
In fact, I’m so committed to this whole cookie thing that I’ve decided to pivot
my entire career and become a Cookie Baking Release Engineer. Feel free to reach
out if you feel this would be important for your business. I am also open to
making donuts and ice cream.