How much faster—and how certain?
A customer asks whether a measured improvement will hold tomorrow. One quick run cannot answer that. We’ll compare repeated experiments, express the size of a difference clearly, and decide whether the variation leaves room for a different conclusion.
Turn independent paired runs into a conditional comparison, import your own Chapter 2 evidence, and plan repetitions that answer the customer’s decision.
01 / The decision
A repeatable benefit is more useful than a winning run
Chapter 5 examined whether the runner measures the intended work. Even with a sound runner, the same request can take different times because of background activity, caches, scheduling or temperature. Variation is this spread of observed results. The next question is whether the improvement is large enough, and consistent enough, to matter.
Suppose a sales operations team is considering a change to its revenue dashboard. Agree on a practical threshold before testing: perhaps at least a 20% reduction in elapsed time for a representative report, with the same answer and service guarantees. A tiny reliable improvement and a large unreliable one support different decisions.
A run is a complete execution of the experimental procedure. Several requests within one warm engine share its state; they are useful observations within that run. They are not automatically separate independent experiments. Restarting the engines helps reset state, but does not remove shared device conditions or time trends.
02 / Worked example
Make the direction and denominator explicit
In one illustrative run, A takes 120 ms and B takes 100 ms. The time ratio A/B is 120 ÷ 100 = 1.20. This says that A took 1.20 times as long, or that B completed the same work at 1.20 times the rate if only that elapsed time matters. B’s time reduction is (120 − 100) ÷ 120 = 16.7%. A 1.20× ratio is therefore not a 20% time reduction.
A 20% time reduction requires B/A ≤ 0.80, equivalently A/B ≥ 1.25. Write the numerator, denominator and unit whenever you use “faster”. Keep startup and warm-query results separate, as in Chapter 2.
The starter evidence contains five independent illustrative pairs: 120/100, 108/100, 132/100, 114/100 and 126/100 ms. Pairing means A and B are tested within the same experimental block under closely matched conditions. Alternate or randomise which goes first; otherwise an order effect can look like an engine effect.
For each pair calculate A/B, then take its natural logarithm,
written ln(A/B). Logs put reciprocal ratios at equal
distances around zero: doubling and halving balance. The average of
those logs is about 0.17981; reversing the log with
exp gives a geometric mean ratio of
1.197. This gives equal weight to each run’s proportional
difference.
How the uncertainty calculation works
The sample standard deviation of the five log ratios is about 0.0793. It describes their spread. Divide by √5 to obtain the estimated standard error, about 0.0355. With five pairs, the Student t critical value is 2.776. The interval is exp(0.17981 ± 2.776 × 0.0355), approximately 1.085–1.321.
The lab uses a two-sided approximate 95% t interval on paired log ratios. The statistical model assumes independent run blocks whose log ratios have a reasonably normal distribution and describe the same stable experiment. With few runs, these assumptions are hard to inspect. For more than 31 runs the lab conservatively retains the 30-degree-of-freedom critical value, 2.042.
The interval lies above parity, 1, for this example. It still crosses the practical threshold of 1.25, so it does not establish the agreed 20% time reduction. A confidence interval concerns the procedure’s coverage over repeated experiments under its assumptions; it is not a promise about 95% of future requests or a 95% probability that a particular product will win.
03 / Evidence lab
Compare run summaries, then inspect the pairs
Read the starter result first, or change one run and analyse again. Try changing B in run 5 from 100 to 150 ms. Observe both the central ratio and the interval: a headline alone hides the less consistent result.
All starter values below are authored illustrative data for learning the method. No database, server or customer was measured to produce them. Editing and analysing these values runs calculations locally in this tab.
Use real Chapter 2 measurements
Run the same Chapter 2 configuration at least five times and download each complete run. Each run creates fresh engines. Retain the same query, row count, repeat count, build and environment. Close unrelated activity, record conditions, and spread or block runs to expose trends. Several files captured seconds apart on one device do not represent every customer deployment.
Select one query, then select all its compatible JSON exports together. A is DuckDB-Wasm and B is PGlite. The importer uses each engine’s median request time within each run; those two medians form one pair. It checks compatible settings, engine identities, build, method and generated input hash, and rejects duplicate run timestamps. It cannot verify independence or equivalent operating conditions for you.
Input and notes save in this browser when storage is available. Downloads include the raw CSV, controls, calculations, notes and any imported original records. Keep a download for your benchmark brief.
04 / Read the evidence
Ask what repetition actually repeated
Inspect the per-run bars before trusting the interval. Are later runs systematically different? Did one option always go second? Was a slow run caused by the tested system or a documented external interruption? Keep every valid run; decide exclusion rules before seeing which option benefits.
Hundreds of requests from one engine session can make a calculation appear precise while leaving session-to-session variability unmeasured. Resampling individual requests as though they were independent runs is inappropriate when they share state. If you use a bootstrap elsewhere, resample independent run blocks and justify its assumptions. This lab does not bootstrap requests.
More runs reduce sampling uncertainty under a stable design. They do not repair unequal work, an unrepresentative workload or a biased timer. Avoid repeatedly adding runs only until an interval excludes 1; choose the number of runs or a valid sequential design in advance.
05 / Independent practice
Write a repeatable comparison plan
Choose one real customer job. State the useful time reduction, what stays fixed, how a run resets state, and how you will pair and order A and B. Choose at least five independent runs for this learning exercise; more may be necessary for your target precision. Keep all raw request records and reduce each run using a declared statistic. Compare a different row count in a separate experiment.
Use the notes field to write a two-sentence finding: “For this job and these conditions, the estimated A/B time ratio was …, with an approximate interval of … . This does/does not resolve our practical threshold because … .” Add the next condition you need to test.
Hint 1 · Define the experimental unit
Describe what happens between two run IDs. If nothing resets or conditions drift continuously, explain why you would treat the blocks as independent.
Hint 2 · Separate statistical and practical questions
Ask both whether the interval crosses 1 and whether it crosses the customer’s useful threshold. These are different questions.
Check your work
The starter ratio is about 1.197, and the interval about 1.085–1.321. The implied central time reduction for B is about 16.5%. The evidence favours B in the illustrative conditions, but does not establish a reduction of at least 20%. Your own numbers should replace every placeholder, with conditions and source attached.
Keep the result attached to its experiment
Download the paired input, original exports, per-run summaries, interval method, ordering, exclusions and notes. Record whether an interval refers to mean log run ratios, mean request latency or another quantity. They answer different questions. Chapter 7 examines the requests that a run summary can conceal.
Further reading
- NIST: confidence limits for the mean explains the t interval and its assumptions.
- Kalibera and Jones: Rigorous Benchmarking in Reasonable Time examines sources of variation and repetition levels in performance experiments.
- Georges, Buytaert and Eeckhout: Statistically Rigorous Java Performance Evaluation discusses warmup and repeated executions in a runtime environment.