ch10 · Three chains, and the binding constraint

Builds on ch09 and ch06.

The question

When three independent chains each demand a different size, which one are you buying?

ch09 followed one chain from stored bytes to hosts. There are two more: one from the requests, one from the working set. The three disagree with each other, and the disagreement is not to be averaged away.

The material

Three chains, sharing a workload and nothing else

A web service has to serve its requests, keep its working set in memory, and hold its records on disk. Those are three requirements with three different arithmetics, and none of them can be derived from the others.

• The request chain runs from the busy hour, through the cost of a request and the margin under the knee, to a number of hosts.

• The memory chain runs from the records held, through the share of them a busy hour touches, to the hosts whose memory can hold that with room to spare.

• The disk chain is ch09’s.

They share the workload and nothing else.

Three spreads, three different shapes, and a great deal of overlap. None of them is the answer.

All three chains are in one file, and it is short enough to read in a sitting:

Buy the largest, not the average, the usual winner or the sum

The count that satisfies all three is the largest of them, and problem 10.1 is that one function call. Spend a minute on the three wrong answers first. Each of them has shipped:

The average. It satisfies at least one chain badly, by construction, and which one depends on the day. A fleet sized between what the requests need and what the working set needs is too small for one of them in every sample where they differ.

The usual winner. Take the memory chain, because it asks for the most more often than either of the others. This is the commonest of the three. It is defensible until somebody asks the model what “more often than either” comes to.

The sum. It buys a fleet for a workload that does not exist. The three chains describe the same hosts doing three things, not three sets of hosts.

How often each one wins

Source — binding-constraint · sizing.evaluate

No chain decides most of the time. The working set wins more often than the others. The request rate is close behind. The disk chain, the one ch09 spent a chapter on, wins least. Then read the three rows after the tie. Size on any one chain alone, even the usual winner, and the fleet is too small more often than not. Wins most often is a fact about a three-way race. Too small is a fact about losing to anybody.

The median gap between the winner and the runner-up, the one in the middle when every draw’s gap is sorted, is large. These are not three estimates of the same thing that differ slightly. They are three different questions with three different answers. The gap at the 95th percentile is larger still.

The last row surprises people. The largest of three uncertain counts sits well above where any one of them usually does, so the fleet the model recommends is bigger than every chain’s typical answer. That is not waste. It is what buying for three requirements at once costs, when each of them is uncertain on its own.

All three chains are in the graph now, meeting at the node that takes the largest. Drag CPU time per request down and watch which chain is in charge change hands.

The shortfall, and the chain that usually wins

Sizing on the chain that wins most often is the natural thing to do, and it is what most sizing does without saying so. Two numbers describe what it costs: how often the fleet is too small because another chain wanted more, and by how much when it is. Problem 10.2 computes both.

Neither is small here, and people do not compute the second. Averaged over every sample, including the ones where the chosen chain was the right one and the shortfall is zero, it looks like a rounding error. Counted only over the samples where the fleet is short, it is not. A chain overtakes another only when its own inputs have gone somewhere unusual, and by the time they have, the gap is wide.

So the honest summary of sizing on one chain is two numbers: how often it is wrong, and how badly when it is. One of them alone is a way of not answering.

Why this gets worse with more chains

The web service has three. A real service has more: a database’s connection limit, a cache’s eviction rate, the network between the hosts, a licence tier. Each is another chance for the answer to be set by something nobody was watching.

The chance that some constraint binds unexpectedly rises with the number of chains, even while the chance of any particular one doing so stays small. A model with six chains, each binding a fraction of the time, spends most of its life with at least one of them unexpectedly in charge.

The observability model in Appendix F has three parallel chains and three separate ceilings for this reason. There is no single number that summarises them, and a model that produced one would be hiding the thing you needed.

What this cannot tell you

Whether there are only three chains. This model has the three somebody thought of. A chain that is not in the model cannot bind inside it, however often it binds outside. That is ch20, and nothing here addresses it.

What a request costs. The request chain rests on a time per request that is an assumption, marked as such in every figure, because no reference machine is declared. A measured one is a rig result nobody has taken, and it is the kind of figure that is quoted from a bench with nothing else running.

Anything about the three chains interacting. They are treated as independent demands on the same hosts. They are not. A working set that no longer fits turns memory reads into disk reads, which raises the cost of a request, which moves the request chain. That is ch08’s regime change running straight through the sizing arithmetic.

Which chain binds for you. The shares above come from one model’s uncertainty over one stated workload. A service that serves small records to many users and one that holds large records for a few are the same model with different inputs and opposite answers.

Problems

Three, in tests/bandwidth_and_the_binding_constraint/. The first two have tests. The last does not, and says why.

10.1 — Three chains, one purchase. One function call. Work out what happens under each of the three obvious wrong answers before writing the right one.

def size_for_all(
    request_hosts: np.ndarray, memory_hosts: np.ndarray, storage_hosts: np.ndarray
) -> np.ndarray:
    """Problem 10.1 - three chains, one purchase.

    Three independent chains each say how many hosts the workload needs: one from how many
    requests arrive at the busy hour, one from how much of the data has to stay in memory, one
    from how much of it has to sit on disk. Return, for every sample, the count that satisfies all
    three.

    It is one function call and the point is which one. Work out what happens under each of the
    obvious wrong answers before you write the right one:

    * take the **average** of the three, and the fleet satisfies none of them;
    * take the **working-set** chain because it wins most often, and you are under-provisioned
      whenever it does not - which the next problem shows is most of the time;
    * **add** them, and you have bought a fleet for a workload nobody has.
    """
    raise NotImplementedError("problem 10.1")

10.2 — The cost of sizing on the chain that usually wins. Compute how often that fleet comes up short, and the median shortfall in those samples only. The first is larger than “usually wins” suggests, and the second is what stops the shortfall looking harmless.

def cost_of_sizing_on_one(
    chosen: np.ndarray, others: tuple[np.ndarray, ...]
) -> tuple[float, float]:
    """Problem 10.2 - what it costs to size on the chain that usually wins.

    Suppose somebody sizes on ``chosen`` alone - the working-set chain, say, because in more of
    the samples than any other it is the one that asks for the most. ``others`` are the chains
    they ignored. Return two numbers:

    * the fraction of samples in which that fleet is **too small**, because some other chain
      wanted strictly more;
    * the median shortfall **in those samples only**, in hosts.

    The first is the one people do not expect. A chain that wins more often than either of the
    others still loses to one of them more often than not, because "wins most often" is a
    statement about a three-way race and "too small" is a statement about losing to anybody. The
    second is the one they do not compute: a shortfall conditional on being short is not a
    rounding error, because the cases where a neglected chain wins are the cases where it wins by
    a lot.
    """
    raise NotImplementedError("problem 10.2")

10.3 — Which chain binds for you. No test: which chain binds depends on quantities only you have.

This chapter has three chains because this model has three. Work out the ones for your system: the quantities that each, on their own, decide how many machines you need. Then work out which binds first.

The useful part is the margin. If one chain binds at twice the other, the second is free capacity you are paying for and nobody is counting. If they bind within a few per cent of each other, your sizing is balanced and also brittle. A small change in either moves which one is in charge, and the argument you rehearsed about the first chain stops applying.

A good answer names at least two chains, says which binds, and by how much. If you can only find one chain, you have found an assumption rather than a fact.

Where to go next

ch11 is the margin that sits under all three chains, and why it is three different margins rather than one.

ch12 puts all of Part III together and produces a number.