ch09 · Capacity

Builds on ch03 and ch04.

The question

How far is what you buy from what you can use?

Further than most people’s first estimate, and always in the same direction: you buy more than you can use. Four terms separate the two, and nobody writes them down together.

The material

The chain

Four terms stand between an application’s storage requirement and a purchase order. Two multiply what you must buy. One divides it. One is a surcharge.

Here is that chain with the rest of the model around it. One node is a colour nothing earlier in the book has had: the measured constant is orange. The ceiling the chain ends in is the third in the model. The constant is the other thing that makes a sizing model, and it would have made this one a sizing model even if no ceiling had.

Replication. Whole copies. Three copies cost three times the space, survive two losses, and are the simplest thing that works. Erasure coding buys the same durability for less space by spreading the data over more pieces, and problem 9.2 is that comparison. Erasure coding is not cleverer, only amortised. It pays for the space it saves with reads that touch more machines, which is a bandwidth problem, and therefore ch10’s.

Compression. The only term that helps you, and the only one that is a measured constant rather than a decision:

Source — web_service_capacity-reference · every input on a slider

Read the last column. That ratio belongs to one codec and one body of data. It does not belong to compression in general, and it does not belong to your data. Take the method rather than the number: point the runner at a sample of your own records and get the ratio that belongs in your model (ch03).

Overhead. Indexes, the write-ahead log, the filesystem’s own bookkeeping: the space the store keeps beside the records so that it can find them and survive a crash. Applied to everything, and larger than people expect once the indexes are counted.

The fill limit. People forget this one because it is not a property of the data at all. You cannot run a disk full, and ch11 is about why the margin is a rule rather than a number. It is in the chain here because the space you hold back is space you still have to buy.

Every term but one makes you buy more

Compression is the only term that helps. It is also the only one measured rather than decided or assumed. Replication and the margin are decisions, and a decision is certain. The overhead is an assumption with a shape, because nobody has counted the indexes. And a measurement over somebody’s corpus carries a standard error, which none of the others do.

A sizing that treats all four as constants is optimistic in the one place that helps. That place is the one whose uncertainty was measured.

Two kinds of terabyte, and the ten per cent

A drive’s datasheet says a trillion bytes. A filesystem counts in powers of two. The difference is about a tenth. Both are called a terabyte in conversation, and a tenth is a large fraction of what compression was going to buy you.

So every node in this book declares a unit, and the toolkit converts rather than assuming. Problem 9.3 is that conversion, and this is the chapter where getting it wrong costs money.

What comes out

Source — web_service_capacity-reference · every input on a slider

Three rows, and each needs a word.

• Raw data is what the disks must hold once every copy, every index and the compression are counted.

• Hosts for storage is how many hosts’ disks that takes. It is this chain’s answer, one of three the model will have by ch10.

• Disk fill at horizon is how full the disks of the fleet somebody bought are at the end of the period, as a fraction of what they can hold. One is full. An interval reaching past one says that in some futures the records do not fit, because the arithmetic carries on past the point where the disks stop.

The interval on the host count spans an order of magnitude. Almost all of that width is the growth rate from ch04, not anything in this chapter’s chain. The disk arithmetic is the well-understood part of the problem. What it is applied to is not.

The measured constant is in the file the same way a ceiling is, and the toolkit reads its stamp rather than its number:

What this cannot tell you

What your data compresses to. The constant above was measured over a synthetic mixture this repository generates, and the mixture’s proportions are an assumption stated in the stamped result. For this figure, that assumption is a larger source of error than the codec, the shard spread, or anything the standard error reports. The number has an uncertainty, and the uncertainty is about the wrong thing.

Anything about record size. The chain above is a chain of bytes. A store holding many small records spends a substantial and sometimes dominant share of its disk on per-record bookkeeping. The overhead term here is a flat multiplier and cannot express that. A model of a service with a small-record problem needs a term this one does not have.

What happens when a host’s disks are lost. Every figure is a healthy fleet. Losing a host means its copies have to be re-made on the survivors, using disk and bandwidth that were doing something else. The chain has no term for the window in which that is happening (ch11).

Whether disk is the chain that binds. It is the chain this chapter followed. It is not the one that most often decides, and the model assumes neither. ch10 is the other two chains, and how often each of the three decides the answer.

Problems

Five, in tests/capacity/. The first four have tests. The last does not, and says why.

9.1 — The chain. The amount you must keep and the three terms that turn it into the disk you buy, one of which divides; the fill limit is left to the node that counts hosts. Getting the division upside down gives an answer wrong by the square of the compression ratio while still looking plausible. Check yours against a case you can do in your head first.

def raw_for(stored: float, replication: float, compression: float, overhead: float) -> float:
    """Problem 9.1 - the chain from what you must keep to what you must buy.

    ``stored`` is the bytes of records the service has to keep. ``replication`` is how many
    copies the store keeps of each. ``compression`` is the ratio the records achieve on disk - a
    two means they halve. ``overhead`` is a multiplier for the indexes and the write-ahead log
    kept beside them, so 1.3 means thirty per cent.

    Return the raw bytes of disk you have to buy.

    Two of those four multiply and one divides, and getting the division the wrong way up gives an
    answer that is wrong by the square of the compression ratio while still looking entirely
    plausible. Work out which is which by asking what each one does to the amount you buy.

    The order matters less than people think and the direction matters more. Check yours against
    a case you can do in your head before running the test.
    """
    raise NotImplementedError("problem 9.1")

9.2 — Erasure coding against copies, at equal safety. Work out the replication factor that survives the same number of losses as a given code, so the two can be compared on space rather than on enthusiasm. Then notice what the saving grows with, and what else grows with it.

def erasure_crossover(data_shards: int, parity_shards: int) -> float:
    """Problem 9.2 - when is erasure coding cheaper than copies?

    Replication keeps whole copies: three copies costs three times the space and survives two
    losses. Erasure coding splits an object into ``data_shards`` pieces and computes
    ``parity_shards`` more, and survives the loss of any ``parity_shards`` of them - for a space
    cost of ``(data + parity) / data``.

    Return the **replication factor** that would give the same durability as this erasure code, so
    that the two can be compared on space at equal safety.

    It is one line and the point is what it exposes. Erasure coding is not cheaper because it is
    cleverer; it is cheaper because it amortises the same protection over more pieces. The saving
    grows with ``data_shards``, and so does the number of machines a single read has to touch -
    which is a bandwidth and latency cost this model has no term for, and ch10 is why that
    matters.
    """
    raise NotImplementedError("problem 9.2")

9.3 — The other kind of terabyte. Re-declare every unit in the model that carries a terabyte in tebibytes, inputs included, and convert each input’s number so that it still means the same bytes. Then check that nothing the model buys has moved: the same hosts, the same money, and every tebibyte figure reading smaller by exactly the ratio. Converting is not compensating, and the difference between the two is the chapter. If a host count moved, a unit was missed, and finding it is the exercise.

def in_binary_units(
    units: dict[str, str], values: dict[str, float | dict]
) -> tuple[dict[str, str], dict[str, float | dict]]:
    """Problem 9.3 - the same model, read in the other kind of terabyte.

    A vendor's TB is a trillion bytes. A filesystem's TiB is 2^40 of them, about ten per cent
    more. Both are spelled "terabyte" in conversation and the difference has bought a lot of
    people a smaller cluster than they thought.

    ``units`` maps every node in the web service model to the unit it declares. ``values`` maps
    every input to its number, or to the band the file writes for it as a dictionary: ``p10``
    and ``p90``, or ``minimum``, ``likely`` and ``maximum``. Return the two, converted. Every
    unit that carries a terabyte, in a numerator or a denominator, ``TB``, ``TB/host``,
    ``USD/TB/month`` and the rest, carries a tebibyte in its place. For a computed node that is a
    relabelling and nothing else: its number is worked out from the inputs and the build converts
    it. For an input it is more than that. An input's number is a claim about bytes, and the same
    number under a new unit is a different claim, so convert the number too: a value, or every
    number in a band, so that it means the same bytes it meant before. A price per
    terabyte-month becomes a slightly higher price per tebibyte-month. Leave every other unit and
    number as it was. The test puts what you return back into the model; no formula changes.

    The test then asserts what only a consistent conversion gives: the model still typechecks,
    nothing the model buys has moved, the same hosts and the same money to the last digit, and
    every node read in tebibytes reads smaller by exactly the ratio. If a host count moved, you
    converted a number wrongly or missed a unit, and which one is the exercise.
    """
    raise NotImplementedError("problem 9.3")

9.4 — Turn it back into a cost model. Name what has to go to make the web service model a cost model: the measured constant this chapter adds, every ceiling before and after it, and everything downstream of them. At least one of its outputs has to survive. Then write one sentence saying what the result can no longer tell anybody. If you cannot name it, you removed something that was doing no work, and the model should not have had it.

def what_to_remove(kinds: dict[str, str], feeds: dict[str, set[str]]) -> list[str]:
    """Problem 9.4 - turn a sizing model back into a cost model, honestly.

    ``scripts/verify-models.py`` classifies a model by what is in it: a ``measured`` node or a
    ``ceiling`` makes it a sizing model, and a model with neither is a cost model whose inputs
    can be sampled. The web service model crossed that line in ch06, when its first ceiling
    arrived; this chapter adds the measured constant that would have crossed it anyway.

    ``kinds`` maps every node in the web service model to its kind: ``input``, ``derived``,
    ``measured`` or ``ceiling``. ``feeds`` maps every node to the names of the nodes its formula
    reads. Return the names to delete so that what is left classifies as a **cost** model and
    still evaluates: every measured constant, every ceiling, and everything downstream of them,
    because a node that reads a deleted node cannot be worked out.

    The test deletes them, outputs included: the ceilings are outputs, and the measured constant
    feeds the recommended host count. At least one of the original outputs has to survive. And
    the point of the exercise is in the third test: having removed them, finish the last line of
    this docstring with one sentence saying what the resulting model can no longer tell anybody.
    If you cannot name it, you have removed something that was not doing any work, and the
    original model should not have had it.

    What it can no longer say:
    """
    raise NotImplementedError("problem 9.4")

9.5 — What your data compresses to. No test: the corpus is your data, and this repository has never seen it.

The constant in this chapter was measured over a synthetic mixture, and the chapter says so. Measure your own: take a real sample of what you store, compress it with the codec you run, at the setting you run it at, and record the ratio and how much you measured.

Then compare it with the figure your capacity plan is currently using, and find out where that figure came from. In this book’s experience it is a vendor’s marketing number, a different codec’s, or nobody remembers.

A good answer has a ratio, a sample size, the codec and its setting, and a sentence about the number it replaces. If your measured ratio matches the planning figure exactly, find out who measured it first. You may have just re-derived a guess.

Where to go next

ch10 is the other two chains, and the question of which of the three you are buying.

Appendix D is the terabyte problem and the rest of the conversions that bite.