Beyond databases
A benchmark travels well only when its definition of success travels with it. Follow an event from sending through processing, then adapt the same reasoning to storage and search.
Define delivery milestones and use a real broker trace to investigate lag, recovery and event correctness.
01 / Customer question
When is an order delivered?
A fulfilment team needs each order processed within a target time while arrivals continue. Kafka stores and carries events. A producer sends them, a broker stores them, and a consumer reads them. The producer’s acknowledgement, consumer receipt and completion of the consumer’s work are separate milestones.
Define the objective before choosing the measurement: for example, 95% of orders processed within two seconds at 20 events per second, with no missing IDs. Check whether that target is met under sustained arrivals and after an interruption.
02 / Worked example
Three clocks for one event
- Offer event ID
- Broker acknowledges
- Consumer receives
- Processing completes
Suppose 100 events are offered over five seconds, all acknowledged, but only 70 are processed during that window. The acknowledgement rate describes broker acceptance; the remaining 30 are still customer work. If the consumer then processes 10 events per second while five arrive per second, the backlog falls by five per second and takes about six seconds to clear. This is an illustrative calculation, not a measured result.
Lag is work waiting behind the producer. Compare
counts at matched times and inspect offsets; a growing gap signals
the consumer is not keeping up. Describe message size, arrival
shape, partitions, producers, consumer groups, batching,
compression, replication and acknowledgement rules. Compare
equivalent guarantees. Here acks=all means the single
in-sync replica, not multiple durable copies.
The harness checks unique IDs, duplicates and receipt order for its one partition. Its processing step is a trace milestone, not a real fulfilment application. It cannot establish ordering across partitions or resilience to broker loss.
03 / Run the experiment
Pause and recover under arrivals
Predict how the gap between offered and processed events changes during a three-second consumer pause. Run this bounded workload and inspect actual event times. The sender is synchronous: if it falls behind schedule, configured rate is not achieved rate.
Download the runnable infrastructure harness · Measurement notes
Unzip the bundle, open a terminal in its
infrastructure directory, and use Docker Compose. First
startup downloads pinned images and builds the client. The trace is
written to results/.
docker compose up -d kafka store db
docker compose run --rm client broker --count 200 --rate 20 --pause-at 2 --pause-for 3
ls results/*.json
For a clean repeat,
docker compose down -v --remove-orphans removes this
project’s service data. Inspect the JSON summary and event
observations, then import the file below. The harness uses one
broker, one partition and one replica: its results describe this
local topology, not a production deployment.
Use the observations to calculate acknowledgement, receipt and completion delays for the same ID. Count offered, processed and errors; chart their difference across time. Compare the gap after resume with the continuing arrival rate. The trace includes received offsets and processing timestamps, allowing a lag proxy; it is not a broker-side consumer-group lag metric. A timeout or error means the run is incomplete.
04 / Inspect evidence
Read a trace
This browser tool inspects evidence; it does not run a broker or pipeline. Import the JSON produced by the harness or paste it below. The table groups observed milestones by elapsed second. Objects and records have different units.
Recorded local functional run · 29 October 2026. This sample shows the method and verification; it does not establish sustainable capacity or production performance.
Download the recorded example · Sample provenance
No evidence loaded.
05 / Practice
Choose a meaningful outcome
A customer compares object stores for a nightly upload and a search service for a product catalogue. Write one success contract for each, including correctness, workload and completion.
Hint 1: What is the user waiting for?
For storage, distinguish acknowledgement from reading the correct bytes; for search, distinguish query response from the time a new product becomes searchable.
Hint 2: What changes the bill or quality?
Storage request size and concurrency affect throughput and request charges. Search requires relevance checks against labelled queries, indexing delay and response-time distributions.
One possible answer
Upload and read back objects of the customer’s size distribution at their concurrency, verify checksums and include request costs. For search, use representative queries and relevance judgments, measure indexing-to-searchable delay and response latency at the expected query rate.
06 / Take it to a customer
Keep the contract with the result
Record topology, versions, workload, achieved arrival rate, acknowledgements, processing completion, errors, lag history, identity checks and pause duration. Repeat with comparable guarantees and report unfinished work. A short clean run proves neither sustained capacity nor fault tolerance.
Further reading: Apache Kafka documentation and Amazon S3 user guide.