Beyond the average
A fast typical response can coexist with frustrated users and unfinished work. We’ll follow individual requests through a busy interval, compare long waits with completed work, and account for every planned request before presenting a service claim.
Define latency and throughput populations, calculate percentiles from raw observations, and expose waiting that a send-only timer can hide.
01 / The decision
What did the customer actually receive?
Chapter 6 compared repeated run summaries. Now imagine that an order screen must respond within 100 ms for at least 99% of requests, with fewer than 1% failures. A mean time cannot establish either promise. You need individual outcomes, a stated clock boundary and the full population of requests that should have received service.
Latency is elapsed time between two named events. Here a request has a planned arrival, an actual send and a completed response or other observed outcome. Throughput is completed work per unit of time. We report successful throughput, sometimes called goodput, separately from offered work and failures.
An error is a completed unsuccessful outcome. A timeout means the caller stopped waiting at its deadline; the server may still be working. Pending work has no terminal outcome at the end of the observation window. Never give a pending request a latency of zero or quietly drop it from the account.
02 / Worked example
Seven quick answers do not mean ten happy users
Our authored trace plans ten requests during a one-second window. Seven finish successfully, one returns an error, one times out and one is still pending. The offered rate is 10 ÷ 1 = 10 requests/s; successful throughput is 7 ÷ 1 = 7 requests/s. The error rate is 10%, the timeout rate is 10%, and the pending fraction is 10%. Their categories sum to all ten planned requests.
Six successful responses take 20 ms after sending. Request r7 is planned at 600 ms, sent at 900 ms and finished at 950 ms. Its send-to-response time is 50 ms, but its planned-arrival-to-response time is 350 ms: 300 ms of waiting before submission plus 50 ms after sending. A customer or upstream queue may experience both.
A percentile describes a sorted population. p50 is the value at or below which at least half the observations fall; p99 uses 99%. This lab uses nearest rank: sort the n values and select position ceil(p × n), counting from 1. With seven successes, p99 is position ceil(0.99 × 7) = 7, the maximum observed value.
The send-only successful times are 20, 20, 20, 20, 20, 20 and 50 ms. Their mean is 170 ÷ 7 = 24.29 ms, p50 is 20 ms and p99 is 50 ms. Including pre-send waiting changes the last value to 350 ms: mean 470 ÷ 7 = 67.14 ms and p99 350 ms. Both summaries still exclude unsuccessful and unfinished work, which must remain visible beside them.
Seven successes do not reliably estimate a production p99. Even 100 observations put the estimate near one of the largest samples, and rare events may not appear at all. Choose enough requests, observation duration and independent repeats to encounter relevant slow paths and maintenance cycles. Do not average per-run p99 values into a combined p99: combine the intended raw population or present the runs separately.
03 / Evidence lab
Change the timing boundary without losing the population
Switch between actual send and planned arrival. Then change r7’s finish time to 990 ms and analyse again. Finally change r10 from pending to a successful completion at 980 ms. Observe the population counts as well as the percentile.
All starter values below are authored illustrative data for learning the method. No database, server or customer was measured to produce them. Editing and analysing these values runs calculations locally in this tab.
Input and notes save in this browser when storage is available. Downloads include the raw CSV, controls, calculations, notes and any imported original records. Keep a download for your benchmark brief.
04 / Measurement traps
A slowing client can hide missing demand
A closed-loop client sends its next request only after the previous one finishes. If each request changes from 10 ms to 100 ms, one such client falls from roughly 100 to 10 requests/s. This may represent a fixed population of users waiting between actions. It does not represent an independent stream of 100 new requests/s.
If arrivals should continue during a stall but the runner waits for responses before scheduling them, the trace omits the requests and waits that would have occurred. This is called coordinated omission. Measuring only actual send times can also omit queueing in the load generator. A planned-arrival timestamp exposes that waiting for requests that were actually planned; it cannot reconstruct requests that were never generated.
An open-loop test schedules arrivals independently of completions. Retain planned timestamps, actual sends, client queue depth, dropped submissions and final outcomes. Verify the generator can keep up. Report the observed arrival schedule and delivery rate; a configured target alone does not establish delivered load.
A timeout at 200 ms establishes that the caller did not receive success within 200 ms. It does not establish the server’s eventual response time. This is a censored observation. Do not turn timeouts into ordinary 200 ms successful responses in the histogram. If late completion matters, continue observing and link it to the original request ID while retaining the user-visible timeout outcome.
05 / Independent practice
Build a service-level account
Design two tests: an interactive lookup and an overnight revenue report. For each, state the work unit, arrival model, timing start and stop, latency population, success criterion, timeout, observation window and unfinished-work policy. An interactive service may need a tail-latency target; a report may need correct completion before a deadline. Explain why each metric serves the decision.
For a real run, retain one row per planned request with an ID and
event timestamps from one monotonic clock, such as
performance.now(). Convert elapsed times to the CSV
format above. Do not compare timestamps from separate machines
without clock synchronisation and an uncertainty bound. Keep
terminal outcomes mutually exclusive in the accounting window.
Use the notes field to challenge this claim: “Our p99 is 50 ms, so we meet the 100 ms requirement.” State the missing population, the hidden waiting and what experiment would resolve them. Download the corrected account with your plan.
Hint 1 · Start with counts
Planned requests = successful + error + timeout + pending for the lab’s window. A server-side retry is additional work, not an extra satisfied customer request.
Hint 2 · Read the boundary
Request r7 waits 900 − 600 = 300 ms before send. The two latency definitions answer different questions.
Check your work
The starter successful throughput is 7/s. The successful-request send-only p99 is 50 ms; planned-arrival p99 is 350 ms. Two of ten planned requests have terminal failures and one remains pending. A claim about all customer requests needs those outcomes as well as a much larger representative sample.
Report the service, its denominator and its limits
Keep raw event records, window length, load model, histogram or sorted observations, percentile method, success/error/timeout/pending counts and measurement boundaries. A percentile with no population or offered rate is incomplete evidence. Chapter 8 uses these measures to find the tested operating range.
Further reading
- Google SRE: Service Level Objectives connects indicators, measurement populations and user needs.
- HdrHistogram: coordinated omission explains omitted observations and why workload generation matters.
- NIST: percentiles discusses quantiles and different estimation conventions.