ch11 · Headroom and failure domains

Builds on ch06 and ch07.

The question

Why is headroom a rule rather than a number?

Because the number is different for every ceiling, for reasons that have nothing to do with each other. And because the moment it becomes a single number, it becomes a number somebody rounds.

The material

Three margins, three different things being protected

Here is every ceiling in the book’s two models, with its declared margin. First the running example’s, as this chapter leaves it, then the observability platform’s:

Source — web_service_headroom-reference · every input on a slider

Source — observability-reference · every input on a slider · 2 constant(s) not yet measured

They are all percentages, and they are not the same kind of thing at all.

A capacity margin protects against a cliff. The disk fills, or the working set stops fitting in memory, and you find out immediately: one as failed writes, the other as a service time that has doubled. What the margin buys is the time between noticing and doing something, plus the space a failed host’s copies need to land in.

A queueing margin protects against a slope. Nothing fails. There is no page. The system slides down ch06’s curve, paying in latency on every request, for as long as nobody looks. This margin is larger, for two reasons. The failure mode is invisible, and recovering from it means adding machines, which ch07 showed works badly.

A scaling margin protects a budget. Nothing fails and nothing gets slow. The fleet costs more than its work is worth. It is the loosest margin in the book, and it is still worth declaring, because a cost that nobody has bounded is a cost that grows.

A single “keep thirty per cent free” rule applied to all three would be too tight for one, too loose for another, and unexplainable for the third. The model file gives a reason beside every margin it declares. Here they are, in its words:

Source — web_service_headroom-reference · every input on a slider

The one margin you can compute

Most headroom is judgement. One piece of it is arithmetic.

A fleet that has to survive losing hosts needs somewhere for those hosts’ work to go. That capacity has to be there beforehand. A fleet that discovers it needs a failure reserve during a failure is already over the knee.

Problem 11.1 is that fraction. It has a consequence people rarely state as a capacity argument. A small fleet pays an enormous margin, because one host in five is a fifth of the fleet. A large fleet pays almost nothing per host. That is a real and quantitative argument for larger failure domains, and it is not the argument people usually give for them.

The second half of that problem is worth more than the arithmetic. The margin is for a loss, not for a failure. A host drained for a kernel upgrade costs exactly the same capacity as one that has died, and planned work is far more common than failure. Most fleets spend their failure reserve on a Tuesday afternoon. A reserve sized for annual hardware failure is not there when somebody starts a rolling upgrade.

The running example carries this as a ceiling of its own: the queueing margin again, audited with one host gone, because a host lost at the busy hour is a queueing problem for the survivors:

  hosts_after_failure:
    kind: derived
    unit: host
    label: hosts left when one dies
    formula: hosts - one_host
  utilisation_after_failure:
    kind: derived
    unit: dimensionless
    label: utilisation with one host down
    formula: busy_cores / (hosts_after_failure * cores_per_host)
  failure_headroom:
    kind: ceiling
    unit: dimensionless
    label: utilisation with one host down
    of: utilisation_after_failure
    limit: 1
    headroom: queueing_margin
    because: >-
      One host down at the busy hour: its share of the requests
      lands on the survivors, and the question is whether they
      are still under the knee. N+1 is this ceiling with the margin
      at zero. The margin here is the queueing margin, because
      a fleet that has lost a host is still a queue (ch11).

Margins do not add

A sizing conversation collects margins. Rebuild wants some. Queueing wants some. Growth between now and the next purchase wants some. Each request arrives separately, each is defensible, and each is granted.

They do not add. Each one takes its share of what the previous one left, so applying them in sequence is multiplication. Three separately modest margins leave you with well under half of the fleet doing the work it was bought for. Problem 11.2 is that composition.

Nobody in the room multiplied them. That is how a fleet ends up twice the size anybody intended, with every individual decision in the chain defensible.

Push the margins up and addition stops describing anything. Three margins of ninety per cent add to nearly three whole fleets, and no system has negative capacity. Taking nine tenths three times over leaves a sliver. That is severe, and at least it is a quantity that exists.

What a margin is for, written down

Every ceiling in this book carries a because. Not because it is tidy, but because the failure mode of a margin is specific and predictable: it gets copied.

A margin with a reason attached can be argued with, adjusted when the reason changes, and dropped when the reason goes away. A margin that is just a number gets carried into the next model, and the one after that, by people who were not in the room. Ten years later an organisation has a thirty-per-cent rule that everybody follows and nobody can source.

The toolkit refuses a ceiling without one. That is the only enforcement available, and it is better than none.

The output a margin produces

Not a verdict. A probability.

Read the last two columns of the tables above. They answer one question: across everything this model thinks could happen, how often does the design end up past this limit? A point estimate comfortably inside the margin tells you about one future only. ch13 is where the other futures come from. ch12 is what the difference between the two readings costs.

What this cannot tell you

Whether any of these margins is right. Every one was declared by somebody, with a reason, and the reason is an argument rather than a measurement. This chapter argues that a margin must exist and must be explicable. It does not argue that these particular ones are correct, and it has no way to.

How long you have. A margin buys time between something going wrong and something being done. How much time depends on how fast your load moves and how quickly anybody notices, and neither is in any model here. A generous margin on a system nobody watches is not generous.

Whether the failure domain is what you think. The failure arithmetic assumes hosts fail independently. They do not. They share racks, power, switches, firmware versions, and the engineer who is applying an update to all of them. A margin sized for one host and spent on a rack is a margin that was not there.

What happens when two margins are needed at once. The composition arithmetic above assumes the margins are for independent things. A host lost during a growth spike during a busy hour is one event, not three, and the model has no term for it.

Problems

Four, in tests/headroom_and_failure_domains/. The first three have tests. The last does not, and says why.

11.1 — What a host loss costs. The one piece of headroom that is arithmetic rather than judgement. Then notice what it says about small fleets, and what it says about planned work.

def failure_reserve(hosts: int, tolerated_losses: int) -> float:
    """Problem 11.1 - the margin a host loss actually costs.

    A fleet of ``hosts`` machines has to survive losing ``tolerated_losses`` of them and still
    serve the busy hour. Return the fraction of the fleet's capacity that has to be kept free for
    that, as a number between zero and one.

    Two things fall out of this that people get wrong in opposite directions.

    A small fleet pays an enormous margin. Losing one host out of five costs a fifth of the
    capacity, and if the margin is not there beforehand its share of the requests lands on
    survivors that are already busy. A large fleet pays almost nothing per host, which is a real
    argument for large failure domains and is not the argument people usually give for them.

    And the margin is for a *loss*, not for a failure. A host that is being drained for a kernel
    upgrade costs exactly the same capacity as one that has died - and planned work is far more
    common than failure, which is why this margin is spent most often on a Tuesday afternoon
    rather than on an incident.
    """
    raise NotImplementedError("problem 11.1")

11.2 — Two margins are not one margin twice. Compose several independent margins. Do not add them; the clue that you cannot is what addition does to three large ones. Then look at what three separately reasonable requests leave you.

def compose(margins: list[float]) -> float:
    """Problem 11.2 - two margins are not one margin twice.

    ``margins`` are independent reasons to keep capacity free: a rebuild reserve, a queueing
    margin, a headroom for the growth between now and the next purchase. Each is a fraction
    between zero and one.

    Return the single fraction of the system that is actually available once all of them are
    applied, expressed as **the margin** - so if a quarter of the system is usable, return 0.75.

    Do not add them. Work out what applying one margin and then another does to what is left, and
    notice that the answer is never as large as the sum, and that for large margins the sum is not
    even a fraction: adding three ninety per cent margins gets you a negative system, which is a
    clue.

    Then look at what a few realistic sets of margins actually leave you, and understand why a
    sizing conversation that treats each margin as a separate reasonable request ends with a
    cluster twice the size anybody intended.
    """
    raise NotImplementedError("problem 11.2")

11.3 — Add a ceiling. The page shows a fragment of the model file: a ceiling on connections per host with its unit, its margin and its reason left empty. Fill them in, and get a verdict and a breach probability out of it. The toolkit refuses it three different ways before it accepts it, and each refusal is a rule this chapter argues for.

# Problem 11.3 - a ceiling on connections per host, declared the way the model file declares one.
#
# The test splices this node into the web service model under `nodes:`, adds it to the outputs,
# and holds it to the three things declaring a ceiling commits you to: a unit the expression
# produces, a margin, and a reason. Three of the fields below are empty. Fill them in; change
# nothing else. The limit is given, and it carries its unit the way the model file carries
# one: multiplied by the constant that holds it.
connections_per_host:
  kind: ceiling
  label: connections per host
  unit:
  of: concurrency / host_count
  limit: 256 * one_request
  headroom:
  because:

11.4 — Your margin, and who chose it. No test: the answer is a number somebody chose, and the interesting part is who.

Find the headroom your system is planned to, then find the person or the document that chose it. This is usually the shortest problem in the book and the most uncomfortable.

Two follow-ups. Does the margin have a reason attached that is not “it is what we have always used”? And does your failure domain match the physical layout? Are the machines you assume fail independently in the same rack, the same power feed, the same availability zone?

A good answer has a number, a name or a document, and a reason. If the reason is round, twenty per cent or thirty per cent, ask what it would have been if the first person to say it had said a different round number. That is usually the whole derivation.

Where to go next

ch12 is Part III assembled: every chain, every margin, and a number at the end of it.

ch13 is what the last two columns of every table in this chapter actually came from.