Don't put all your customers in one basket

Every growing system has a reassuring moment. You add another server, then another, and the whole thing feels sturdier. More machines, more safety. Surely.

Then one of them fails at 2am, and a startling number of customers go dark with it.

Bigger feels safer

Most of us, me included, relax a little as a system grows. It's a natural instinct, and it's only half right.

A big system rarely fails all at once. It fails one piece at a time. So the question that matters is how many customers sit behind each piece. If nobody set a limit, the answer is "however many happened to land there", and over time some pieces quietly collect far more than their share.

A big team learned it the hard way

In September, InfoQ reported that Uber had redesigned how its M3DB database spreads data, splitting it into smaller subclusters to limit how much one failure can affect. They have more machines than most of us will ever see. They still decided the answer was smaller fenced-off groups.

Eggs in boxes of six

The rule: put a fixed limit on how many customers one failure can reach, and keep it fixed as you grow.

Think of eggs. One huge basket holds everything, and one trip ruins the lot. Boxes of six cost a bit more cardboard, but a dropped box costs you six eggs at most, however many boxes you own.

Software works the same way. Split customers into small groups, each looked after by its own few machines. When something fails, only that group notices. Everyone else carries on.

Here's the bit that surprises people. With one basket, growing doesn't help: a failure still takes the same share of everyone. With boxes, every new box makes any single failure a smaller share of the whole.

Keep the shuffling inside the box

Systems move data around to balance the load. That's routine, as long as the move stays inside one group. Moving data between groups is a bigger job and deserves the care of one: a deliberate decision and a quiet moment, never an automatic reflex.

Otherwise the fix for one busy group can drag several healthy ones into the problem.

What it costs

More groups means more to look after: more things to watch, more places to check during an outage. Groups that are too small also have less spare room when a machine fails.

There's no perfect size. The useful step is to write down one number: if one group goes dark, how many customers notice? Then look at it again every time your customer count doubles.

The basket felt efficient right up to the moment you tripped.