Memory Ordering on Real Hardware

28 · Memory Ordering on Real Hardware¶

The question¶

What do four cores cost each other, and what does a fence actually buy?

ch20 established what a lock is made of and said plainly that what it costs is contention, which that target had no way to express. This chapter has four real cores, and it is also where the reader meets a second memory model — which is the reason this book uses two architectures.

The material¶

A cost with no shared data¶

Two threads, two counters, no shared data and no race. One of the two arrangements below costs more than twice what the other does.

Conditions: target host · aarch64 cross compiler, any machine — nothing here was executed · aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 · the book's portable flags · 2026-09-17 · Source: bench/results/sharing-layout.json, code hash 76da3c773e2fc037.

The counters are the same counters. What differs is padding, and therefore whether the two land on the same cache line. The compiler and the layout settle that before anything runs, which is why this figure is measured here rather than on the board.

Coherence works in lines, not in variables. When one core writes a line, every other core’s copy of that line is invalidated, and a core that wanted a different variable on that line has to fetch it back. Neither thread is sharing anything; they are fighting over a container that happens to hold both their things.

The name for this is false sharing, and the name is slightly misleading. The sharing is real — the line is genuinely shared — and what is false is the implication that the program meant to share anything.

Conditions: target host · Raspberry Pi 5 Model B Rev 1.1 · Linux 6.18.50+rpt-rpi-2712 · cc (Debian 14.2.0-19) 14.2.0 · -O2 -g -Wall -Wextra · 2026-09-16 · Source: bench/results/sharing-host.json, code hash 7070f0b86e84fccc.

The gap is widest at two threads and narrows as cores are added: more contenders do not make false sharing worse, which is the opposite of what a scaling curve would suggest. Two caches trading one line back and forth, with nothing else to overlap against, is already the expensive case.

Problem 28.1 is the analysis: given where the fields are and who writes them, which pairs will contend. The case worth getting right is two fields on one line written by the same thread, which costs nothing, because sharing a line is not what is at fault.

The same need, two spellings¶

Now the second memory model, put beside the first on purpose.

ch20 printed what a release looks like on both architectures, and the two are not alike. RISC-V emits fence rw,w and then an ordinary store: an instruction between the two things being ordered. AArch64 emits stlr — a store that carries the ordering itself. One instruction, no fence, and a reader who learned that a barrier is something you put between two operations will not recognise it as a barrier at all.

Both satisfy the same requirement. Neither spelling is memory ordering itself.

That is why this book uses two architectures: a reader shown one weak memory model concludes that model is memory ordering. Shown two, they find that “weakly ordered” is a family; that a fence is an architecture-specific spelling of an architecture-independent need; and that what actually transfers is the mechanism underneath — store buffers, coherence, and the fact that another core can observe your writes in an order you did not write them in.

Problem 28.3 is that correspondence as a table, and what matters is the asymmetry between the two spellings rather than the table itself.

What ordering costs¶

Conditions: target host · Raspberry Pi 5 Model B Rev 1.1 · Linux 6.18.50+rpt-rpi-2712 · cc (Debian 14.2.0-19) 14.2.0 · -O2 -g -Wall -Wextra · 2026-09-16 · Source: bench/results/sharing-host.json, code hash 7070f0b86e84fccc.

Two columns because the difference between them is most of what there is to know. An atomic operation on a line nobody else wants is nearly free: the core already owns the line exclusively and the operation is local. The same operation on a line four cores are all writing costs the coherence traffic to take the line away from whoever had it, every time — and the table puts that at more than an order of magnitude.

So “atomics are expensive” is not a fact about atomics. It is a fact about contention, and the same instruction has two costs that differ by a large factor depending on something that is not in the instruction.

Now read the rows against each other, because they answer the second half of this chapter’s question. Uncontended, the ordered atomic costs measurably more than the relaxed one: that difference is what the ordering buys, and it is the only place in this book where a fence has a price of its own. Contended, the two are the same to within the width of the measurement. The ordering is free exactly when the contention is not, because once the line has to be fetched from another core, waiting for it has already done everything the ordering was going to.

Which makes the plain row the one to be careful with. An ordinary non-atomic increment is cheaper than either atomic in both columns, and on four cores it is cheaper by a very large factor — and it is also wrong, for ch20’s reason. That row is what the lost update costs you: nothing, until it costs you everything.

Predict before you measure¶

ch24 asked for a distribution instead of a number. This chapter asks for something else first: a prediction.

Amdahl’s law says what a scaling curve can look like at best. If a quarter of the work cannot be parallelised, four cores buy about two and a quarter times, and no number of cores buys four ever. That is a ceiling computed from the program, before the machine is involved at all.

A measured curve on its own tells you very little. A measured curve that falls short of a predicted one tells you there is something to find — and false sharing, atomic contention and coherence traffic are exactly the things that live in the gap. Problem 28.2 is the prediction. This chapter’s tables give two thread counts rather than a curve, which is enough to show the direction and not enough to fit anything to; the curve is yours to take, on your own machine, and the prediction is what makes it worth taking.

What we measured¶

Where the fields land: that decides whether the cores will fight, and finding it needs no machine. The fights themselves were then measured on the board — two layouts at two thread counts — along with three kinds of increment, each on one core and on four.

The runner refuses to stamp a layout in which the packed structure’s counters have stopped sharing a line or the padded one’s have started, because either would leave this chapter demonstrating nothing while still producing a table.

What this cannot tell you¶

Anything about a different interconnect. The header says four cores and this coherence fabric, and it means it. The scaling curve is this board’s; the mechanism — invalidate on write, fetch on read, in units of lines — is not.

Whether your program has false sharing. The analysis here is over a layout you can see. Real programs share lines through allocators, through arrays of small structures handed one per thread, and through padding decisions made in libraries. Finding it in something you did not write is ch30’s equipment.

What the memory model permits. This chapter measures what the machine does, and a weak memory model is a statement about what it is allowed to do. Those are different, and the difference is dangerous: a reordering that never happens on this chip may be permitted, and code that relies on not seeing it is broken on a chip that does. ch20’s problem 20.2 is about the permission, deliberately, and no measurement can replace it.

The cost of getting it wrong. Every fence in this chapter is correct. What an incorrect one costs is not a number — it is a program that works for a year and then does not, on a machine nobody has yet.

Problems¶

Three, in tests/memory_ordering_on_real_hardware/sharing.c.

28.1 — Which fields will two cores fight over? Given offsets and writers, count the contending pairs. Two fields on one line written by the same thread are not one of them.

python3 -m pytest tests/memory_ordering_on_real_hardware/test_problem_1_pairs.py

28.2 — What does the curve look like before you measure it? Amdahl’s law, applied. Do this before the board reports: a prediction you fail to reach is information, and a measurement with nothing to compare it against is much less.

python3 -m pytest tests/memory_ordering_on_real_hardware/test_problem_2_scaling.py

28.3 — Same requirement, two architectures. Give the spelling each one uses. The table is small; the observation that one of them has no separate instruction at all is the reason the problem exists.

python3 -m pytest tests/memory_ordering_on_real_hardware/test_problem_3_spelling.py

Where to go next¶

The ARM architecture reference manual’s memory-model chapter Arm Ltd (n.d.) and the RISC-V unprivileged specification’s RVWMO chapter RISC-V International (n.d.) are worth reading in the same sitting, in either order. They describe the same kind of object with different vocabulary, and the correspondence is much easier to see when the two are half an hour apart than when they are years.

ch29 returns to Part IV with the same equipment. Everything xv6 demonstrated structurally — a system call, a page fault, a context switch — has a price on this machine, and the chapter puts the two accounts side by side.