Can you trust your benchmark?

01 / How demand reaches the system

A waiting client changes the offered load

Offered load is the work the test intends to introduce. Achieved work is what actually completes during observation. Throughput is a count divided by time, such as correct responses per second. Always name which count you use: submissions, acknowledgements and verified completions are different events.

A closed-loop client sends a request, waits for the response, then starts the next. If the database or client slows down, fewer requests are offered. This can fit a person who waits for a page before continuing, but a single closed-loop client does not represent many independent arrivals.

A scheduled-arrival test intends requests at stated times even while earlier work is waiting. Here we schedule twelve arrivals, five milliseconds apart: 0, 5, 10, …, 55 ms. A single client then drains that queue in order, with at most one SQL request in flight. This is a finite demonstration of waiting and client limitations, not a sustained capacity test.

In a browser, timers can fire late. The record therefore keeps both the intended arrival and when the page actually enqueued it. A delay before enqueue indicates timer or scheduling lateness. A gap between enqueue and submission indicates waiting for the client plus its preparation. The database query timer alone includes neither of those earlier waits.

02 / A worked timeline

The missing time can be before the SQL

Consider an invented timeline for explanation. A request is intended at 10 ms, enqueued at 12 ms, submitted at 32 ms, and its response arrives at 34 ms. The query request took 2 ms. The elapsed time from the intended arrival was 24 ms.

Intended arrival       10 ms
Actually enqueued      12 ms    2 ms scheduler lateness
SQL submitted          32 ms   20 ms queue + client preparation
Full response received 34 ms    2 ms measured query request

Total from intended arrival = 34 − 10 = 24 ms
Total = scheduler lateness + queue/preparation + request time

The arithmetic describes one work item. A chart of only the 2 ms query response would omit its earlier delay. An application measurement may legitimately start at submission, but it should not be presented as the customer's whole wait when requests have already been waiting elsewhere.

A related problem is coordinated omission: a load generator that waits for a slow response stops generating the arrivals that would otherwise have waited during the slowdown. Its apparently healthy latency distribution may leave out those affected requests. Match the arrival model to the scenario and document where your clock begins; adding more decimals cannot repair the missing population.

03 / Investigate the runner

Make the client the bottleneck

A bottleneck is the stage that limits progress under the tested conditions. Start with one engine, closed-loop arrivals and no intentional client pause. Download the result. Add a 20 ms client pause before every SQL submission, then run again. Finally switch to scheduled arrivals while retaining that pause. The query and 20,000-row fixture stay identical.

-- Twenty thousand deterministic orders, as in Chapter 2.
SELECT order_id, month_id, region_id, amount_cents
FROM orders WHERE order_id = 10000;
-- Twelve requests, one connection, at most one request in flight.

04 / Test the explanation

A faster query can leave throughput unchanged

With a 20 ms pause, the client can submit at most about 50 requests per second even if each database request were instantaneous. That is a mathematical upper bound from the deliberate pause, not a measured engine limit. Real timer delays and query work reduce the achieved rate further. The intervention gives you a cause to test: removing the pause should reduce client delay, though short observations still vary.

Under the closed loop, the next intended arrival moves later whenever a previous request takes longer. Under scheduled arrivals, the intended timeline remains fixed and queued work can accumulate. Compare total time from intended arrival with query request time for the same row. A large difference locates delay outside the query boundary; it does not prove the database is slow.

These twelve requests are too short to establish sustained capacity or tail latency. A finite replay drains its queue, so zero pending work at the end does not show that an ongoing arrival stream is stable. A sustained test needs a fixed observation period, continuing arrivals, bounded backlog, errors, resource observations and response targets measured together. Do not hide late requests by ending the clock when submissions stop.

05 / Observe the right lifecycle

Warm operation is only one phase

Startup can include downloading engine files, compiling WebAssembly and opening the database. Warm-up gives code and caches an initial request. Sustained operation must last long enough to encounter relevant background tasks. Recovery asks what happens after disruption. Each phase answers a different customer question; choose and report the boundary before comparing numbers.

This lab records startup, loading and statistics separately and warms the lookup once. It does not assert that one warm-up reaches a stable state. Cached downloads can make later startup runs different. Server checkpoints, garbage collection, compaction, backups and recovery are absent from this short in-memory exercise. For an application used once after each import, include startup and preparation in a separate whole-job test; for an always-on service, design sustained and recovery observations.

06 / Independent practice

Challenge a plausible benchmark claim

A colleague writes: “Our database handles the peak rate. We sent the next request only after the previous one returned. The successful responses were quick, and we excluded timeouts from the chart.” Treat this as a constructed review exercise. Write what is missing, then propose an experiment that could resolve it.

1. Collect the three runner records above. Compare query time, time before submission, achieved work and all outcome counts.
2. Explain why one waiting client does not establish an independent arrival rate. Describe how excluding timeouts changes the population being reported.
3. Write a sustained-test protocol: arrival schedule, observation duration, drain policy, response target, error accounting, client resource checks and expected correctness.

For an executable extension, change the scheduled interval in runner() in src/methods-runtime.ts from 5 ms to a named constant, then use that constant in the intended timestamps as well as the scheduling expression. Update the visible description and run both 5 ms and 25 ms intervals locally. This is a useful first harness edit: the intended arrival record must change with the schedule, or the evidence becomes misleading. Keep twelve requests initially so you can inspect every timeline.

Draft and controls save on this device when storage is available. Your current notes are attached to the original completed record when downloaded.

07 / Sources and reproducibility

Keep the observer in the evidence

All clocks use the page's monotonic performance.now(), which measures elapsed time without depending on wall-clock changes. Timestamps share one clock and are offsets from observation start. A submitted request includes worker messages, engine work and conversion of the full result. Intended-to-completed time additionally includes scheduler lateness, client queueing and the intentional pause. No clock synchronisation across machines is involved.

Limits are 60 seconds for startup, 30 seconds for load/setup, and 15 seconds for a request. Stop, hidden-tab detection, page exit and worker errors terminate workers. An incomplete record is explicitly labelled; do not use its partial samples as a successful comparison. This runner offers no configurable request deadline or server cancellation contract beyond worker termination.

• Postgres pgbench documentation explains rate scheduling, schedule lag, latency limits and transaction counts in a production-oriented benchmark tool.
• wrk2's description of coordinated omission explains the consequences of measuring only after actual submission.
• The browser monotonic clock and timer scheduling limits explain relevant browser behaviour.
• Measurement contract and reproduction; engine source and licence notices.

With a defined workload and an inspected runner, repeated runs become meaningful. The next chapter asks how large a difference is and how much uncertainty remains.