The Same Program on Both Targets

23 · The Same Program on Both Targets¶

The question¶

What does watching a program in a debugger fail to tell me about what it costs?

Everything up to here has built a complete structural account of a machine. You can say what a program compiles to, where its data sits, how its address space is built, what it costs the kernel to ask for anything, and how many blocks of disk traffic a single byte becomes. All of it was obtained on a target chosen precisely because you can stop it anywhere and look.

None of it is a duration, and this chapter is about what that leaves out.

The material¶

One program, two routes¶

Here is a program small enough that the parts before this one explain it completely.

An array of cells, each holding a number and the index of another cell. The chain visits every cell exactly once. Two routes add up every number: one walks the array in order, and the other follows the chain.

  long sysfs_bridge_sequential(const struct sysfs_cell *cells, long count) {  \
    long total = 0;                                                           \
    for (long i = 0; i < count; i++)                                          \
      total += cells[i].value;                                                \
    return total;                                                             \
  }                                                                           \
                                                                              \
  /* Every value, in the order the chain gives them. Same values, same total, \
   * same number of additions, and one more load to find out where to go. */  \

They add the same numbers in a different order and return the same total. Compiled for the xv6 target and for the reference machine, run on both, they agree:

Conditions: target xv6 · riscv64 cross build, any machine — run under user-mode QEMU for counts only · riscv64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 · xv6's own CFLAGS on one side; the book's portable flags on the other · 2026-09-17 · Source: bench/results/bridge-both.json, code hash 1ab750e4b1e975e9.

That agreement is what the book has established so far. The same source produces the same answer on two machines with different kernels, different instruction sets and different everything else — because ch13 checked the data model matched, and because the answer is a property of the arithmetic rather than of the machine.

What the structure predicts¶

Now the instructions. RISC-V first:

sysfs_bridge_sequential compiled for riscv64 at -O2 — the xv6 target’s instruction set, 12 instructions.

0000000000000052 <sysfs_bridge_sequential>:
  52:	blez	a1,6c <.L16>
  56:	slli	a1,a1,0x4
  58:	mv	a5,a0
  5a:	add	a3,a0,a1
  5e:	li	a0,0

0000000000000060 <.L15>:
  60:	ld	a4,0(a5)
  62:	addi	a5,a5,16
  64:	add	a0,a0,a4
  66:	bne	a5,a3,60 <.L15>
  6a:	ret

000000000000006c <.L16>:
  6c:	li	a0,0
  6e:	ret

Conditions: target xv6 · riscv64 cross compiler, any machine — nothing here was executed · riscv64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 · -O2 -Wall -Wextra -march=rv64gc -mabi=lp64d -c · 2026-09-17 · Source: bench/results/bridge-riscv64.json, code hash 2c0d5645ba9b7b66.

sysfs_bridge_sequential compiled for aarch64 at -O2 — the host target’s instruction set, 13 instructions.

0000000000000090 <sysfs_bridge_sequential>:
  90:	cmp	x1, #0x0
  94:	b.le	bc <sysfs_bridge_sequential+0x2c>
  98:	mov	x2, x0
  9c:	mov	x0, #0x0                   	// #0
  a0:	add	x3, x2, x1, lsl #4
  a4:	nop
  a8:	ldr	x1, [x2], #16
  ac:	add	x0, x0, x1
  b0:	cmp	x2, x3
  b4:	b.ne	a8 <sysfs_bridge_sequential+0x18>  // b.any
  b8:	ret
  bc:	mov	x0, #0x0                   	// #0
  c0:	ret

Conditions: target host · aarch64 cross compiler, any machine — nothing here was executed · aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 · -O2 -Wall -Wextra -c · 2026-09-17 · Source: bench/results/bridge-aarch64.json, code hash 140cc3a3926d377c.

sysfs_bridge_chased compiled for riscv64 at -O2 — the xv6 target’s instruction set, 15 instructions.

0000000000000070 <sysfs_bridge_chased>:
  70:	mv	a2,a0
  72:	blez	a1,8e <.L21>
  76:	li	a4,0
  78:	li	a5,0
  7a:	li	a0,0

000000000000007c <.L20>:
  7c:	slli	a5,a5,0x4
  7e:	add	a5,a5,a2
  80:	ld	a3,0(a5)
  82:	addi	a4,a4,1
  84:	ld	a5,8(a5)
  86:	add	a0,a0,a3
  88:	bne	a1,a4,7c <.L20>
  8c:	ret

000000000000008e <.L21>:
  8e:	li	a0,0
  90:	ret

Conditions: target xv6 · riscv64 cross compiler, any machine — nothing here was executed · riscv64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 · -O2 -Wall -Wextra -march=rv64gc -mabi=lp64d -c · 2026-09-17 · Source: bench/results/bridge-riscv64.json, code hash 2c0d5645ba9b7b66.

sysfs_bridge_chased compiled for aarch64 at -O2 — the host target’s instruction set, 18 instructions.

00000000000000c4 <sysfs_bridge_chased>:
  c4:	mov	x4, x0
  c8:	cmp	x1, #0x0
  cc:	b.le	104 <sysfs_bridge_chased+0x40>
  d0:	mov	x3, #0x0                   	// #0
  d4:	mov	x2, #0x0                   	// #0
  d8:	mov	x0, #0x0                   	// #0
  dc:	nop
  e0:	lsl	x2, x2, #4
  e4:	add	x3, x3, #0x1
  e8:	add	x6, x4, x2
  ec:	ldr	x5, [x4, x2]
  f0:	ldr	x2, [x6, #8]
  f4:	add	x0, x0, x5
  f8:	cmp	x1, x3
  fc:	b.ne	e0 <sysfs_bridge_chased+0x1c>  // b.any
 100:	ret
 104:	mov	x0, #0x0                   	// #0
 108:	ret

Conditions: target host · aarch64 cross compiler, any machine — nothing here was executed · aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 · -O2 -Wall -Wextra -c · 2026-09-17 · Source: bench/results/bridge-aarch64.json, code hash 140cc3a3926d377c.

Read the loop bodies. The sequential route loads a value, advances a pointer by a constant, adds, and branches. The chased route does the same and then loads again to find out where to go next, and has to compute an address from an index rather than stepping.

So: one more load, and a shift and an add. Everything Part III and Part IV have to say about these two functions is in that sentence, and it predicts that the chased route costs somewhat under twice what the sequential one does.

Hold on to that prediction. It is the most that a complete structural understanding can offer, it was arrived at correctly, and Part V exists because of how wrong it is.

The number the first target could not supply¶

Conditions: target host · Raspberry Pi 5 Model B Rev 1.1 · Linux 6.18.50+rpt-rpi-2712 · cc (Debian 14.2.0-19) 14.2.0 · -O2 -g -Wall -Wextra · 2026-09-16 · Source: bench/results/bridge-host.json, code hash 248a965fb6193f75.

That table came from the board and could have come from nowhere else — and it is the one place in this book where a duration appears before you have been given any way to judge one. Take the two numbers on trust for exactly one chapter: ch24 is next precisely because it is about how much to trust them, and it comes back to put a spread and an interval around figures like these. A duration measured under QEMU would describe the laptop that ran the emulator: it has no cache to miss, no prefetcher to defeat, no store buffer to fill, and no memory that takes time. It would produce a number, the number would look like the others in this book, and it would mean nothing at all. So until the board answered, this table was a box saying so.

Read the last two rows against each other. The first is the prediction the section above arrived at, from the instruction counts; the second is what the machine did. Everything Part III and Part IV can say about these two functions was in that prediction, and it was arrived at correctly, and the machine disagrees with it by more than the whole of the structural account can explain.

The chased route’s whole cost is that each load has to finish before the next address is known. That is a statement about a memory system, and the target that has taught you everything else in this book does not have one. ch25 is where that cost gets taken apart.

Three things differ at once¶

The board’s numbers will be easy to misattribute, and avoiding that is the skill the rest of Part V is built on.

The two runs differ in three ways simultaneously. One is emulated and one is not. One is xv6 and one is Linux. One is RISC-V and one is AArch64. A difference in the measurement could be caused by any of them, and the temptation — universal, and not confined to beginners — is to attribute it to whichever is most interesting.

The fix is not care. It is design: compare two configurations that differ in exactly one thing. xv6 under QEMU against Linux under QEMU isolates the kernel. Linux under QEMU against Linux on hardware isolates the emulation. Linux on one board against Linux on another isolates the instruction set. Nothing can be isolated by comparing the first of those with the last, and no amount of statistics applied afterwards recovers what the design gave away.

Problems 23.1 and 23.2 are that move, mechanically, until it is automatic.

What survives the crossing¶

It is easy to summarise everything behind you as “structure transfers and cost does not”, and that is wrong in both directions.

The instruction count is structural and does not transfer. Two instruction sets do not emit the same number of instructions for the same source — the table above shows they do not, for this very program. A model built on counting instructions is a model of one compiler for one architecture.

The memory layout is structural and does transfer, but not automatically: it transfers because both targets are LP64 — long and pointers are both eight bytes — with the same alignment rules, which ch13 measured rather than assumed. A book that had chosen a 32-bit target for Part III would have had to say something quite different here.

And one thing transfers that is neither: the mechanisms themselves. A page fault is a page fault, a context switch saves the callee-saved registers, a log makes a group of writes atomic. What changes across the crossing is every number attached to them, which is exactly why Part V is arranged as Part IV’s chapters asked again.

What we measured¶

The answer both routes compute, on both targets, and the instructions each compiles to on both architectures — all of it reproducible, none of it a duration. The one figure this chapter is actually about — the cost of the two routes — has been measured on the board.

What the earlier parts predict about these two functions does not depend on that measurement, and neither does the argument about confounds; the measurement supplies the size of the error and not its existence.

What this cannot tell you¶

Why the prediction was wrong. The central claim here — that a complete structural model predicts the cost badly — is shown in one table and explained in none of this chapter. The next eight are the explanation.

Which of the three differences matters most. Naming the confound is not separating it. Doing that needs configurations this book does not ship: a Linux image for the RISC-V target, or a second board. ch24 starts from the machine the book does have.

Whether the structural model is useless. It is not, and the chapter would be dishonest to imply it. It predicts the answer exactly, predicts the layout exactly, and predicts which of two programs does more work — which is often the question. What it cannot do is turn that into a ratio, and knowing which questions your model answers is worth more than a better model.

Problems¶

Three, in tests/the_same_program_on_both_targets/crossing.c. All three are graded against a table of six configurations that the tests read for themselves, so there is no key to find.

23.1 — What does comparing these two configurations isolate? All thirty-six pairs. A comparison isolates a variable only when it is the single thing that differs; otherwise it confounds, and the pair this book’s own two targets form confounds all three.

python3 -m pytest tests/the_same_program_on_both_targets/test_problem_1_isolates.py

23.2 — Which configuration would test this claim? Somebody says the difference is caused by the emulator, or the kernel, or the instruction set. Given where you are, name the configuration that would settle it — or say that none of the six will, which happens and is the right answer when it does.

python3 -m pytest tests/the_same_program_on_both_targets/test_problem_2_pair.py

23.3 — Does this observation transfer? Six things you could observe about the program. Sorting them into structure and cost gets two of them wrong.

python3 -m pytest tests/the_same_program_on_both_targets/test_problem_3_transfers.py

Where to go next¶

Part V. Every chapter of it names the earlier chapter whose cost it measures, and the first of them is about the instrument rather than the machine: before measuring anything, ch24 asks how you would know a measurement was wrong, which on a board that throttles under sustained load is not a rhetorical question.

For the confound itself, Mytkowicz and colleagues Mytkowicz et al. (2009) is the paper to read — it shows measured speedups appearing and disappearing according to the size of an environment variable, which is a more alarming demonstration of this chapter’s point than anything here.