Benchmark the complete pipeline

01 / Customer question

When is the upload usable?

A sales team submits order files throughout the day. They need all records, correctly transformed and queryable, within a time target at their expected arrival rate. An upload acknowledgement only confirms an intermediate step. Define the record set and transformation before the run, then follow each unique run, object and record ID.

The Python client, JSON serialization and Compose network are part of this tested system. The ingestion adapter is triggered synchronously after each upload; it reads the object and publishes its records. The sink writes each value doubled, then commits the Kafka offset. A database upsert makes repeated keys idempotent; duplicate receipts are still counted. Producer retries are bounded; other failures are recorded and not silently retried.

02 / Worked example

Measure the customer’s wait directly

Imagine 20 objects with five records each, submitted at two objects per second. For each object, start at upload submission and stop at the first query that finds all five expected doubled values. A 200 ms polling interval adds up to approximately one polling interval of detection delay, plus query cost. This is per-object completion, not per-record latency. The harness records both the polling count and verification time.

All trace timestamps use the same process monotonic clock; no cross-host clock synchronization is needed for these observations. In a distributed instrumented deployment, disclose clock synchronization and uncertainty. Never add stage p99 values to invent an end-to-end p99: stage delays overlap and the slowest records may differ.

Compare compatible units. If each object has five records, two objects per second means ten records per second. A sink processing eight records per second will accumulate work even if the database alone handles more. The illustrative rates are not measured service results.

03 / Run and diagnose

Slow the sink, then restore it

Run the baseline below. Repeat with --sink-delay 0.05 to add 50 ms to each sink write, keeping the other settings fixed. The pause occurs while the sender continues. Compare the growth and drain of the gap, verified object times, errors, and completeness. A constraint change tests a bottleneck explanation; it does not prove a production system will behave the same way.

Download the runnable infrastructure harness · Measurement notes

Unzip the bundle, open a terminal in its infrastructure directory, and use Docker Compose. First startup downloads pinned images and builds the client. The trace is written to results/.

docker compose up -d kafka store db
docker compose run --rm client pipeline --count 20 --records 5 --rate 2 --pause-at 2 --pause-for 3
ls results/*.json

For a clean repeat, docker compose down -v --remove-orphans removes this project’s service data. Inspect the JSON summary and event observations, then import the file below. The harness uses one broker, one partition and one replica: its results describe this local topology, not a production deployment.

Check actual offered timestamps; the synchronous ingestion client may fall behind its schedule. The verification query detects missing or incorrect values, and the receipt trace detects duplicate deliveries. A unique database key prevents duplicate rows, so absence of duplicate rows alone is not proof of exactly-once delivery. Timed-out objects are unfinished. Collect logs and repeat runs before making a capacity claim.

04 / Inspect evidence

Read a trace

This browser tool inspects evidence; it does not run a broker or pipeline. Import the JSON produced by the harness or paste it below. The table groups observed milestones by elapsed second. Objects and records have different units.

05 / Practice

Test a bottleneck hypothesis

The broker acknowledges quickly but queryable completion gets later on every object. What would you measure and change before recommending a larger database?

06 / Take it to a customer

Report the whole path

Keep the commands, configuration, topology, achieved input and verified output, failures, queue history, recovery, identity checks, polling interval and query cost. Vary object sizes, records, bursts, batching, transformations and concurrent queries to match the customer. This reference harness fixes some of these choices; extend it or use a production-shaped test before generalizing.

Further reading: Kafka documentation, PostgreSQL documentation and SeaweedFS documentation.