Present a comparison you can defend
Turn a customer question into a repeatable experiment and an evidence-backed recommendation. Build a brief, run a self-checking workload, preserve the conditions and explain exactly where the finding applies. Then use the same method to review somebody else’s claim.
A portable project containing the benchmark brief, runnable setup, correctness checks, run artifacts, analysis, capacity assumptions and a customer-facing report. You can use a database, broker or complete pipeline. The builder saves your draft and packages evidence; it does not decide whether your claims are true.
01 / Begin with a decision
Agree what would change the customer’s mind
Chapter 11 connected performance evidence to a deployment and its cost. The final task is to make that chain inspectable. Imagine a sales operations team choosing a way to serve its order screen and revenue dashboard. “Which database is faster?” is too open to guide the purchase. “Which tested deployment meets our workload, correctness, response-time, recovery and cost targets?” leads to an experiment.
A benchmark brief is the agreement you write before running: the decision, customer workload, alternatives, acceptance criteria and method. An acceptance criterion is an observable pass/fail condition, such as no missing or duplicate order IDs, p99 client response time below 200 ms at a stated arrival rate, bounded pending work, and recovery within ten minutes while new work continues. The numbers are customer requirements to agree, not defaults this book prescribes.
Choose the smallest experiment that can affect the decision. If the only evidence is Chapter 2, ask about the tested DuckDB-Wasm and PGlite builds on that browser and dataset. If the decision concerns server deployment capacity, use the native/service exercises and record the real topology. If the outcome crosses several services, measure its full path as in Chapter 10.
A decision and falsifiable acceptance criteria define the work.
Checked outputs, raw observations and exact setup preserve what happened.
A conditional recommendation states scope, uncertainty and next action.
02 / Worked example
Rewrite a broad claim into an auditable finding
Suppose an invented worksheet says: “A is twice as fast as B.” Its raw records contain five paired browser runs of one aggregation, all with checked totals. The run-level medians are A: 18, 19, 20, 21, 22 ms and B: 36, 38, 40, 42, 44 ms. These numbers are a teaching example, not a measurement of either engine in this book.
The median of the run medians is 20 ms for A and 40 ms for B, a 2× ratio for that timing boundary. This does not mean a 2× throughput increase, a 2× improvement on every query or half the infrastructure cost. Requests within the same run share conditions; they are not automatically independent experiments. Use Chapter 6’s paired analysis and intervals when making a measured magnitude claim.
“In the five illustrative paired runs of this checked aggregation, A’s median run time was 20 ms and B’s was 40 ms. The finding applies to the saved dataset, builds, configuration and browser timing boundary. We have not established server capacity, concurrent response times or operating cost. The next test repeats the customer’s mixed workload against the proposed server topology.”
For an actual customer report, replace the invented values with your observations, cite the run IDs and show the independent-run uncertainty. A concise recommendation starts with the decision it informs, then explains the result and its limits. A customer can act on a conditional finding if the remaining uncertainty and next action are clear.
| Claim component | Evidence to retain |
|---|---|
| Equivalent useful work | Generator/query revision, expected answers or IDs, observed checks and guarantees. |
| Performance difference | Raw samples, independent run IDs, timing boundary, order, warmup, analysis and variability. |
| Customer relevance | Workload source, arrival pattern, data size/skew and agreed acceptance criteria. |
| Deployment recommendation | Server topology, complete-path and failure tests, dated regional cost model and sensitivity. |
03 / A reproducible starting runner
Read, run and change a self-checking experiment
A runner is the program that prepares work, submits it, checks answers and writes observations. You have already used one through the browser controls. Now take control of a small script. Download the Python HTTP comparison runner. It uses Python 3.10 or newer with no added packages and includes two tiny local HTTP services. An HTTP request sends the same integer array to each; each returns its sum.
Open a terminal, a window where you type commands. Change to the
download folder with cd. Run the following in three
terminals, keeping the first two open while the third compares them.
On systems that use python rather than
python3, substitute that executable.
# Terminal 1: start A, an explicit addition loop
python3 decision-runner.py serve --port 8011 --variant loop
# Terminal 2: start B, Python's built-in sum
python3 decision-runner.py serve --port 8012 --variant builtin
# Terminal 3: submit, verify and preserve observations
python3 decision-runner.py compare --count 1000 --repeats 5 --requests 20 --seed 42 --output comparison.json
The service listens only on your own machine. Stop it with Ctrl+C
after the comparison. The --count argument sets how
many integers each request contains; --repeats repeats
whole A/B blocks; --requests sets requests per option
per block; --seed makes the random block order
repeatable. A JSON file is structured text you can open and inspect.
Attach the resulting comparison.json in the report
builder.
The expected sum is computed independently as
n × (n − 1) / 2. The client measures from HTTP
submission through response parsing and the correctness check using
one monotonic clock. It records warmups, errors and unfinished
requests separately. A request timeout defaults to five seconds.
Incorrect answers and interrupted/failed runs leave an artifact and
return a nonzero process exit status. Service startup is outside the
timed region.
This runner is one sequential, closed-loop client: it waits for a response before submitting the next request. It teaches correctness, boundaries, repeated blocks and artifact writing. It does not generate a fixed arrival rate or establish sustainable server capacity. Loopback HTTP and client work may dominate the tiny sum operation. Separate blocks share the same processes; use fresh process launches for independent startup experiments and record that change.
-
Run the script as supplied. Inspect
method,samplesand per-repetition summaries, including failures. -
Change
--countto 10,000 while keeping the other arguments fixed. State whether the observed difference belongs to the service computation, network/serialization path or a combination you have not isolated. - Make a copy of the script. Change one service to return the wrong sum. Confirm the runner reports incorrect answers and exits unsuccessfully. Keep this negative control artifact.
- Replace the request function with your customer operation, preserve an independent correctness check, and record the revision and exact command. Revisit the arrival model before using it for a capacity question.
For a database, broker or complete-pipeline capstone
You can begin with the real broker harness or complete pipeline harness instead. Keep their pinned setup, configuration and verification commands. Record the deployed resources and actual verified rate, latency, backlog, errors and recovery. The local single-replica reference deployment teaches the method; it does not establish production availability. Use a tested topology that matches the purchase decision.
A reproducibility package another person can use
project/
brief.md # question, criteria, plan before running
source-revision.txt # exact commit and uncommitted changes
setup-and-reset.txt # prerequisites, versions, topology, commands
configuration/ # actual settings and generated workload
raw/ # all runs, including failures
checks/ # oracle and verification results
analysis/ # commands that regenerate the summaries
capacity-plan.json # sources, assumptions, scenario exports
benchmark-project.json # report text and evidence register
benchmark-report.md # concise recommendationUse relative artifact paths and checksums for large files. An evidence link is useful only if the reviewer can access the artifact. Keep sensitive customer data out of shareable exports; use an approved representative dataset and describe its differences.
04 / Your report workshop
Connect the brief, observations and recommendation
Write the customer question and acceptance criteria first. Fill the setup and runner fields before execution so another person can follow your procedure. After running, attach the records and cite their evidence IDs, such as E1, next to numerical findings. Retain failed and incomplete runs; explain exclusions rather than deleting inconvenient observations.
Attach a Chapter 2 download to retain its original fixture, engine versions, phases, samples and interpretation. It is labelled browser evidence. Infrastructure JSON records and Chapter 11 capacity plans receive their own labels. The builder embeds parsed JSON and records a SHA-256 checksum of the originally uploaded file bytes; a checksum helps identify an artifact, but does not validate its contents. Add references for source code, raw traces, dated prices or other artifacts that live elsewhere.
The local draft belongs to this browser and origin. Download project JSON for backup: it contains the text, evidence register and embedded JSON. Markdown contains the readable report and references, so share the project JSON and external artifacts alongside it. Import restores a project; attach evidence adds a run to the existing project. Text completeness is not a technical review or automatic certification.
05 / Independent capstone
Design, run, recommend and challenge
Choose a real decision from your work: a database query path, a broker delivery guarantee, an object-storage pattern, search quality/latency or an end-to-end ingestion workflow. Start with one customer outcome and two equivalent alternatives. Write the brief before the experiment. Submit both the recommendation and the artifacts that could disprove it.
- Agree: name the customer question, workload source, criteria, guarantees, decision owner and constraints.
- Implement: build or adapt the runner, independent expected-output check, reset procedure and recorded setup. Prove a deliberately wrong result is detected.
- Run: collect independent repetitions with a defined order, sufficient duration, original conditions, all outcomes and failure accounting. Check the client and service for bottlenecks.
- Analyse: regenerate the summary from raw records. Include distribution, independent-run variation, useful output and unresolved uncertainty. Link capacity/TCO claims only to suitable deployment evidence.
- Recommend: write one paragraph with the decision, measured finding, scope, tradeoff and next action. Give a reviewer the package and ask them to reproduce one result from the instructions.
Hint 1 · Make the question testable
“Can this topology keep every uploaded record queryable within the agreed end-to-end target under this arrival trace, including the defined worker outage?” names an outcome, conditions and a possible failure. Add the actual numerical targets and expected record set.
Hint 2 · Follow a claim backwards
For every number in your recommendation, point to the raw record, analysis command, timing boundary and correctness check. If one is missing, label the number as an assumption or defer the claim.
Hint 3 · Decide what to test next
Choose the uncertainty most likely to reverse the recommendation. A native server repetition may matter more than another browser timing. A recovery test may matter more than a larger steady-state peak. Explain the decision each would resolve.
Review a published comparison
Select a public benchmark report and record its URL, author, publication date and exact claim. You may use TPC-H published results and the associated disclosure reports as a starting point, or a report in your customer’s field. Read the detailed methodology as well as the headline. This exercise does not assume the report is wrong.
Identify the workload, data size, versions, topology, guarantees, client/load model, timing points, correctness checks, failures, repetitions, prices and commercial context that it actually supplies. Separate “not disclosed” from “known to be unfair.” Then propose one fair follow-up: preserve equivalent work and guarantees, change one relevant variable, define success in advance, and state which decision would change. Put that review in the builder’s final field.
06 / A recommendation with boundaries
Leave the reviewer a clear route to the evidence
Your final paragraph should answer the customer’s question at the tested conditions. Explain when the result could reverse, which evidence is missing and the next action. Distinguish measured facts, external claims and assumptions. Report useful outcomes alongside failure and missed-target counts, and explain the practical cost of each.
A second reader should be able to recreate the fixture, run both options, detect a wrong result, collect an artifact and regenerate at least one finding without private instructions from you. That is the capstone’s acceptance test. The builder can organise the package; a real reproduction and critical review establish whether it is usable.
For further reading, use the Transaction Processing Performance Council’s benchmark specifications to study formal workload and disclosure rules, the SPEC fair-use rules for examples of comparison scope and publication requirements, and the Sizing and TCO companion to deepen the cost and uncertainty analysis. Each benchmark’s rules apply to its own results; your customer still needs a workload and acceptance contract relevant to their decision.