What happens as demand grows?
A quiet test can choose the wrong system for a busy month. We’ll separate data growth from incoming demand, inspect where completed work falls behind, and decide what operating range the evidence supports before adding resources.
Find passing load points from reconciled evidence, explain the difference between rate and concurrency, and design a controlled experiment that tests a suspected bottleneck.
01 / The decision
How much work can the deployment sustain?
Chapter 7 defined latency, successful throughput and unfinished work. A customer now expects demand to double. Sustainable capacity is the work rate a specified deployment can maintain while meeting its correctness, latency and failure requirements, with bounded queues over a relevant duration. The highest completed rate from a short burst does not establish it.
Keep three growth questions separate. Data size changes how much stored information a query examines. Arrival rate changes how many new requests arrive per second. Concurrency is how many requests are in progress at once. A sweep repeats a test at several values of one variable while controlling the others.
A larger data table may change scans and cache behaviour. More arrivals can create queues even with an unchanged table. Raising a closed-loop client count changes the number of waiting users, rather than directly setting an arrival rate. Run separate sweeps first so that the cause of a change is interpretable.
02 / Worked example
Find the range that passes the customer’s target
Consider two illustrative deployment options with the same correct workload and service guarantees. The target is successful-request p99 ≤ 150 ms, errors plus timeouts ≤ 1% of offered requests, and no growth in pending work. Each supplied observation lasts 60 seconds, starts empty and uses a fixed intended arrival rate.
At 200 requests/s, A receives 12,000 arrivals and completes all 12,000 correctly, with p99 120 ms and no failures or pending work. Its successful throughput is 12,000 ÷ 60 = 200/s. This point passes the screen.
At the same offered rate, B completes 11,400 successes, reaches 300 timeouts and ends with 300 pending requests. The account reconciles: 12,000 = 11,400 + 300 + 300. Successful throughput is 190/s, timeout fraction is 2.5%, and pending growth is 300 ÷ 60 = 5/s. Its supplied p99 of 180 ms is also above target. A good mean time would not rescue this point.
At 300/s, A achieves 250 successful requests/s, but p99 reaches 400 ms and pending work grows. Its highest successful throughput is therefore above its highest passing supplied load. The supplied screen yields A at 200/s and B at 100/s. It does not prove their precise capacity limits, the behaviour between points, or stability across hours.
Relate concurrency to time without confusing cause and effect
For a stable system, Little’s law relates average work in progress L, throughput λ and average time in system W: L = λW. At 100 completed requests/s and an average 0.05 seconds in the same system boundary, average work in progress is 5 requests. Units cancel: requests/s × seconds = requests.
Use means and matching populations over a stable interval. p99 cannot replace the mean in this equation. Growing backlog violates the steady-state premise, and changing the number of clients can change both throughput and time. The identity does not predict that doubling clients will double capacity.
03 / Evidence lab
Change the target and inspect the constraint
Analyse the supplied steps, then relax the p99 target to 200 ms. B’s 200/s point still fails because timeouts and pending work exceed the rule. Inspect the counts before editing them: every arrival must end as success, error, timeout or pending in this accounting boundary.
All starter values below are authored illustrative data for learning the method. No database, server or customer was measured to produce them. Editing and analysing these values runs calculations locally in this tab.
Input and notes save in this browser when storage is available. Downloads include the raw CSV, controls, calculations, notes and any imported original records. Keep a download for your benchmark brief.
Server and client CPU columns are context for hypotheses, not proof of a bottleneck. The p99 is a supplied successful-request summary; the original request records and percentile method are required for a real report. A timeout leaves this caller’s pending count even if server work continues, so measure server queues separately.
04 / Diagnose
Test a bottleneck explanation with one controlled change
A bottleneck is a constraint that limits the useful work of the tested path. Near-full server CPU with growing queues suggests CPU pressure; it does not establish it. CPU sampling can miss bursts, a single saturated core can hide within a low whole-host average, and the runner or network may limit arrival delivery.
State a testable explanation: “At 200/s, B is limited by server CPU for this query mix.” Repeat a baseline around 150–200/s, then change only the available CPU while holding data, query mix, durability, memory, client capacity and versions fixed. Measure the actual delivered rate, server CPU and queues, client scheduling delay, failures and latency. Alternate baseline and changed configurations to expose time trends.
If extra CPU moves the passing range and reduces the relevant queue under otherwise comparable conditions, that supports the explanation. If it does not, inspect plans, storage, lock waits, network and generator headroom before adding more resources. Adding memory and CPU while also removing persistence would prevent you from attributing the improvement.
A crossover is a point where the ordering of the options changes as a variable grows. Two isolated measurements do not locate it. Add closely spaced points around the suspected region, repeat them, and retain uncertainty as in Chapter 6. An option can have lower latency at low demand while another supports a higher sustainable rate.
05 / Run and design
Build a growth experiment you can defend
First run a real data-size sweep using Chapter 2’s database lab. Keep the query and repeat count fixed; collect at least five fresh run exports at 2,000 rows and five at 20,000 rows. Analyse each size separately in Chapter 6. This measures data growth for these browser builds, with one request at a time. Record the two sizes as tested points rather than projecting a line to a million rows.
For a real service load sweep, use the runnable infrastructure harness and its declared service configuration introduced in Chapters 9–10. Its configured arrival rate is a target: verify actual request timestamps, matched output and queue growth in every run. Do not convert its broker or pipeline records into database capacity claims. To test databases under concurrent demand, run an equivalent native/server workload and record its driver, deployment and guarantees.
- Choose a customer job, representative data and correctness check. Define a p99 target, allowed failure fraction, backlog boundary and duration before seeing results.
- Start at a low rate. Increase one variable at a time, retain actual arrivals and completions, and verify the generator is keeping up. Observe long enough to include relevant background work.
- Repeat closely spaced points near a failure or crossover. Test recovery while arrivals continue; a queue that drains only after arrivals stop does not demonstrate sustainable capacity.
- Change one suspected resource constraint, repeat baseline and changed configurations, and compare complete service outcomes.
In your notes, recommend a tested operating range and state headroom separately. For example, choosing 150/s below a repeatedly verified 200/s operating point uses 25% of that tested rate as margin. It does not prove that every burst or failure can be absorbed. Chapter 11 turns verified service capacity and explicit headroom into a deployment and cost comparison.
Hint 1 · Reconcile before interpreting
For each lab row: starting pending + rate × seconds = success + error + timeout + ending pending. The lab rejects rows that cannot satisfy this identity.
Hint 2 · Distinguish a target from delivered demand
A configured 300/s runner may send only 180/s if it stalls. Keep intended arrivals, sent requests and completed work. Report dropped or queued submissions.
Check your work
A’s supplied passing points are 100/s and 200/s. B’s only passing supplied point is 100/s. A’s 300/s point completes 250 successes/s but fails latency, failure and pending-growth requirements. Relaxing latency alone does not make B’s 200/s point pass. Neither option has a measured precise maximum from these three illustrative steps.
Keep the curve and the conditions
Retain the offered and achieved rates, duration, per-request outcomes, queue observations, resource measurements, configuration and repeat IDs for every point. State the changed variable and what stayed fixed. A short screen identifies points worth repeating; a deployment capacity claim needs representative sustained measurements and recovery checks.
Further reading
- Google SRE: Handling Overload explains queue growth, overload and useful service.
- Little: A Proof for the Queuing Formula L = λW provides the foundation for relating average time, throughput and work in progress.
- Brendan Gregg: the USE Method structures resource investigation around utilisation, saturation and errors.