Why my releases start with a rehearsal
You know the moment. It's ten to five on a Friday, the fix is small, every check has passed, and somebody sends it live the way you'd reach for the kettle.
(The kettle has never taken down a website. This is different.)
Putting new software live is called a release, and on most teams it's one command. One command that does everything: copies the new code onto the real server, changes the settings, updates the data. That's lovely until it runs with the wrong version, or a setting gets lost on the way, or nobody reads what it says back.
The release did exactly what it was told. Nobody meant to tell it.
Yes, I've taken the shortcut too
I once pushed a fix straight to the copy of the code that goes live, skipping the step where someone looks it over first. The fix was tested. It was, if I say so myself, a perfectly good fix.
I was told, quite correctly, that the route was wrong even though the fix was right. That stung for about a day. Then it became a rule: nothing goes live until I've signed it off myself. Going in through the fire exit gets you into the building, but the alarm still has a point.
A release that starts with a rehearsal
The rule underneath is older than computers: let the safe thing happen by default, and make the risky thing take a deliberate step. So for the software I'm building, every release now starts with a rehearsal.
The rehearsal goes through every step without changing anything. It looks at the live server, compares it with what's about to arrive, and writes a report. Nothing moves.
The real release needs a second, deliberate act: typing a confirmation word. A button is one slip of muscle memory away from a very different afternoon.
Yes, it's fussy. So is a seatbelt. You pay two seconds every trip for the one day you'd have regretted it.
My software serves many customers from one system. One bad release lands on all of them at once, the way a train that goes to the wrong station takes every passenger with it.
Three questions the rehearsal answers
A rehearsal that only says "OK" is a builder shouting "should be fine" from the van. A good one answers the questions you'd ask a colleague before they touched anything live.
- What exactly will change? Every file added, changed or removed, listed. A summary won't do.
- Is this the version we tested? If it isn't, the rehearsal says so and stops.
- What would make the real release refuse? Every reason, listed, before anything runs.
The last one earns its keep. A refusal you read in a report is a to-do item. The same refusal halfway through the real thing is an emergency, and you're in it.
Know where "undo" stops working
The other half of the rule is the undo button. If something goes wrong, the system can put the old version back on its own. That's right, up to a point.
Once the stored customer data has been reshaped to fit the new version, the old version can't read it any more. Putting it back can do more damage than the original problem. It's like putting the old key back in a lock you've already changed.
So decide in advance where the automatic undo stops and a person takes over, and write it down. Before that point, the system fixes itself. After it, the site shows a "back soon" page, and the system tells a human exactly where it stopped and which backup to use.
What it costs
It isn't free. Every release now runs twice, and an urgent fix waits a few extra minutes for a report someone has to read. Under pressure, a team will start asking for a "skip the rehearsal" switch.
Don't build it. The urgent fix is the release most likely to go wrong, because nobody had time to think about it. If the rehearsal's too slow to live with, make it faster and keep it.
None of this makes releases safe. It just means the dangerous bit only happens when somebody chooses it.
The kettle, for the record, is still undefeated.