ch20 · The missing node

Builds on ch14 and ch19.

The question

How do you find the error that no amount of sampling can see?

Every chapter since ch13 has deferred this one. It is the limitation of the entire method. It deserves its own chapter rather than a paragraph at the end of somebody else’s.

The material

Two ways to be wrong

Wrong about a number. An input is off. The model is the right shape and the arithmetic is right, but one of the quantities is not what you thought. Every chapter so far has been about this error. ch13 quantifies it, ch14 refines it, and ch19 says which one to go and fix.

Wrong about the shape. A cost line is missing. A chain is not in the model. A ceiling nobody declared. Two things multiplied that should have been added.

Sampling handles the first perfectly, and it is blind to the second. Worse than blind: it produces a beautifully converged interval around the wrong answer, and the convergence looks like rigour.

Four things that could be wrong, and the book can count two

That split matters more than any other in the book. Here it is again with all four pieces named, because the book has been using them for twenty chapters without putting them in one place.

A measurement wobbles. A measured constant has a standard error, and it is as likely to be high as low (ch03). This is the smallest of the four, and the only one anybody can reduce by working harder.

A number is unknown. An input nobody measured, given a distribution somebody chose: a price, a growth rate, a count of label values (ch13). This is usually the largest thing inside the interval, and ch19 is about which of them to go after.

Those two are what the interval is made of. Both are quantities the model can carry, and the whole apparatus of Parts IV and VI is about them.

The world takes a different path. A launch doubles the busy hour. Growth stops. Somebody halves how long records are kept. This one is not in the interval, and it is why this book has scenarios at all. A scenario is a second run rather than a wider distribution, because the alternative is not a value the current model could have produced. A fleet bought for the growth case is a different model of the world, not an unlucky draw from this one (ch12, ch16).

The model is the wrong shape. This is the error this chapter is about. It appears in no interval and no scenario, because nothing in the file knows the term is missing.

Naming the four says what a wide interval is and is not. A wide interval is a report about the first two. It is silent about the third, which is a modelling decision somebody took. It is silent about the fourth, which is a modelling decision nobody knew they were taking.

The book’s own example, on a published page

Source — observability-reference · every input on a slider · 2 constant(s) not yet measured

The observability model has an ingest figure that is metrics plus logs. It is arithmetically correct. Every input feeding it is declared, sourced and sampled, and the interval around it is as honest as the rest of the book.

It is also missing an entire chain, because nobody has measured spans per request:

Source — observability-reference · every input on a slider · 2 constant(s) not yet measured

This one is visible, because the model file format happens to have a node for the missing thing, and the toolkit marks it. That is the easy case. It is on a published page so that you have something to compare the hard case against.

The hard case is a chain nobody thought of at all. It has no node, so there is nothing to mark, no box to render, and no way for the model to know it is incomplete. The output is a number with an interval around it, and no indication whatsoever that it is a lower bound.

Why the interval makes it worse

A single number invites doubt. An interval does not. It looks as if the doubt has already been accounted for.

So a structurally incomplete model with a tight interval is more dangerous than the same model with no interval at all. The apparatus that was supposed to communicate uncertainty ends up communicating false confidence. It quantified the uncertainty it could see, and said nothing about the rest.

That is why every chapter in this book has a What this cannot tell you section. It is also why the one in a chapter with a model in it must name what the structure omits, not only what the inputs are uncertain about. That is the only defence available, and it is a weak one.

What works

Nothing automatic. Four things that are not:

Compare against an invoice. The strongest test available. A model of something that already exists can be checked against what it cost, and that number is a fact the model did not have. Problem 20.3 is that check, and the only exercise in the book where the oracle is outside the model.

Ask what is not in the graph. Read the node list as a list of categories, and ask what category is absent. The web service model has no line for rack space, cross-connects, backup, the database’s own licence, or the migration that fills the fleet. Each of those is obvious once named, and invisible until then.

Distrust an answer that is too neat. A total that is too round. A unit cost suspiciously close to a supplier’s headline price. A utilisation that is exactly what somebody hoped. Each is a reason to look for the term that is missing, not a reason to celebrate.

Get somebody who did not build it to read it. The single most effective technique, and the least automatable. A model’s author cannot see its missing chain, because the same gap in their thinking produced both.

The two wrong responses, and one of them is measured

Problem 20.1 is the first trap. An observation falls outside the interval. Is the model refuted?

Almost certainly not, on one observation. A 90% interval is supposed to be missed one time in ten. A rule that rejects on a single miss will therefore reject a correct model sooner or later, and the more observations you make, the surer that becomes. Deciding what would count as evidence is harder than it looks. The problem makes you state a rule rather than react.

Problem 20.2 is the second trap, and it is the one that gets shipped. The model disagrees with reality, so make the model vaguer until it stops disagreeing. Widen the inputs. The observation lands inside, and everyone relaxes.

Measure what that costs and the answer is unambiguous. The interval grows until the model can no longer tell two designs apart, which is the only thing it was for. The median does not move. So the headline number is unchanged and only the doubt has grown, which is why it passes review.

A model that cannot be wrong has stopped being able to be useful. Widening is how a model becomes unfalsifiable. An unfalsifiable model is an expensive way of writing down what somebody already believed.

What this cannot tell you

Whether this chapter’s own model is complete. It is not. The observability model has no line for the network between tiers, no term for the cost of a query nobody ran, and no notion of the people who operate it. Those are the omissions somebody noticed. The chapter’s argument is that there are others, and that nothing in the repository can find them.

How likely a missing node is. There is no distribution over “things nobody thought of”. Any attempt to quantify structural uncertainty ends up as another model with its own missing pieces.

Whether the four techniques above are enough. They are what this book has. Three of them need something outside the model, and the fourth needs somebody outside the team. That is a fair summary of the limitation: a model cannot audit itself. Every technique that works is one that brings in information the model did not have.

When to stop looking. There is no test that says a model is complete. There is only a model that has been compared against reality at least once, and a model that never has been. The second is a model whose structure nobody has checked, however good its interval looks.

Problems

Four, in tests/the_missing_node/. The first three have tests. The last does not, and says why.

20.1 — What would count as evidence? Some observations fall outside the interval. Decide what it would take to call the model refuted, state a rule that weighs how far out they fall against how many you have, and implement it over the whole set. One miss is not it.

def is_refuted(samples: np.ndarray, observed: np.ndarray) -> bool:
    """Problem 20.1 - what would count as evidence against the model?

    ``samples`` is a model's output distribution. ``observed`` is every figure you have from the
    real system the model was supposed to describe, one per observation: a single number in an
    array of one, a year of invoices in an array of twelve.

    Return whether the set refutes the model.

    The whole problem is in the word "refuted", and the trap is that a 90% interval is *supposed*
    to be missed one time in ten. One observation outside it is not evidence; it is the expected
    behaviour of an interval that is doing its job. A model rejected on that basis would be
    rejected roughly whenever it was right.

    Two things bear on the verdict, and a rule has to weigh both: how much of the model's belief
    lies beyond each observation, and how many observations you have. A figure the model puts a
    twentieth of its belief beyond is unremarkable once and damning fifty times over; a figure it
    puts none of its belief beyond is damning on its own. So a rule is a threshold on how
    surprising the whole set would be if the model were right. State one that holds together, and
    implement it.

    The tests check the cases at both ends: a single miss just outside the interval is never a
    refutation, and an observation far enough out that no plausible model produces it always is,
    however few of them you have. In between, the tests only check that your rule is monotonic
    in both directions - never less damning with more copies of the same figure, never less
    damning further out - because there is no right answer there and pretending otherwise would
    be the same error the chapter is about.
    """
    raise NotImplementedError("problem 20.1")

20.2 — The wrong repair, measured. Widen the model until it agrees with the observation, then measure what the interval has become. A model that cannot be wrong has stopped being able to be useful, and this is what that costs.

def widen_until_it_fits(samples: np.ndarray, observation: float) -> float:
    """Problem 20.2 - the wrong repair, measured.

    A model disagrees with an observation. The easiest response is to make the model vaguer until
    it stops disagreeing.

    Return the factor by which the distribution's spread about its median would have to grow for
    ``observation`` to land inside the 90% interval. Scale the samples about their median; do not
    shift them. Scaling the output's own draws stands in for widening the inputs, which is what
    somebody would do to the file: the same repair, one step downstream. The factor is at least
    one, because an observation already inside needs no widening.

    Then look at what that does to the interval. The model now agrees with the observation and can
    no longer distinguish between designs, which is the only thing it was for. A model that cannot
    be wrong has stopped being able to be useful, and this number is what that costs.
    """
    raise NotImplementedError("problem 20.2")

20.3 — Find the missing node. The page shows a copy of a model file that is deliberately incomplete, and its stated interval is a lie: an invented monthly figure, the average of twelve invoices, falls outside it. Repair the model so the observation lands inside the interval. The inputs it has stay as they are, and the repair is a quantity the model never mentioned and the line that prices it. There is no answer key; the oracle is a figure the model does not contain.

# Problem 20.3 - the copy of the fixture you repair. The page shows this file; the test loads it
# and compares it with fixtures/model.yaml, the incomplete model as shipped, which stays as it is.
#
# An exercise fixture for problem 20.3, and deliberately incomplete.
#
# This is NOT one of the book's models and nothing in it is measured. It is a small monthly cost
# model for a hosted service, of the kind somebody would write in an afternoon, and it is missing
# exactly one cost line. Every number in it is invented for the exercise; the file lives under
# tests/ rather than models/ so that it can never be mistaken for a figure the book publishes.
#
# Your repair goes in this file, the model's own way: a node with a unit, a formula or a band,
# and a provenance source, plus the line that prices what it names.
model: incomplete_service
title: A hosted service, monthly (exercise fixture — incomplete)
currency: USD

nodes:
  instances:
    kind: input
    unit: node
    value: 40
    provenance: {kind: assumption, source: "exercise fixture"}
    range: [10, 200]

  instance_price:
    kind: input
    unit: USD/node/month
    distribution: {lognormal: {p10: 210.0, p90: 340.0}}
    provenance: {kind: assumption, source: "exercise fixture"}

  compute_cost:
    kind: derived
    unit: USD/month
    formula: instances * instance_price

  storage_tb:
    kind: input
    unit: TB
    distribution: {triangular: {minimum: 300.0, likely: 480.0, maximum: 900.0}}
    provenance: {kind: assumption, source: "exercise fixture"}

  storage_price:
    kind: input
    unit: USD/TB/month
    distribution: {lognormal: {p10: 16.0, p90: 27.0}}
    provenance: {kind: assumption, source: "exercise fixture"}

  storage_cost:
    kind: derived
    unit: USD/month
    formula: storage_tb * storage_price

  support_rate:
    kind: input
    unit: dimensionless
    distribution: {triangular: {minimum: 0.06, likely: 0.1, maximum: 0.16}}
    provenance: {kind: assumption, source: "exercise fixture"}

  support_cost:
    kind: derived
    unit: USD/month
    formula: (compute_cost + storage_cost) * support_rate

  monthly_cost:
    kind: derived
    unit: USD/month
    formula: compute_cost + storage_cost + support_cost

outputs:
  - monthly_cost

20.4 — The node you have not written down. No test: if a check could find what is missing, the chapter would not be necessary.

Take your own model and go looking for what is missing. Use whichever of this chapter’s techniques applies: compare it against a system that already exists, look for a quantity that appears in an invoice and nowhere in the model, or ask somebody who operates it rather than plans it.

The last one is usually the fastest. The people who carry a system know about the quantity that doubles the storage and appears in no design document. They are rarely asked.

A good answer names at least one quantity that was not in the model and says how much it moves the answer. What would show it wrong is adding the quantity and watching no output move: then it was missing and harmless, and the one that matters is still missing. If you find nothing, the honest conclusion is that you have not looked hard enough, not that the model is complete. There is no test for completeness, which is what this chapter is about.

Where to go next

ch21 is the last chapter of the argument, and this one is its prerequisite. An honest presentation of a total includes what the model does not contain, and saying so out loud is harder than any of the arithmetic that came before it.

ch14 ends on the sentence this chapter expands, if you skipped it: a model whose answer has stopped moving between runs has settled its arithmetic, and nothing else.