Dispatch

A last-mile delivery telemetry pipeline, and the thing it found.

Kafka → Spark Structured Streaming → a bronze/silver/gold lake → a star schema, orchestrated by Airflow, with dbt marts and eight blocking data-quality gates that stop a run before a bad load reaches the warehouse.

125,264 events
20,000 trips
21 partitions
121 tests
0 containers needed

The finding

The delivery promise uses one global speed for every store. The stores do not move at one speed, so the error it leaves behind is more than twice as large at one of them, and a single pooled average reports that as one problem.

See the numbers →

Architecture

The system, box by box. Every component links to the file and line that implements it, pinned to the commit it was generated from.

Results

What the pipeline found, generated from the warehouse at build time. Per-store promise error, daily breach rates, and every gate verdict.

Handoff

Dispatch measures how wrong a flat delivery promise is. Arrival predicts what it should have been. fct_trip is the entire contract between them, diagrammed.

What each piece is doing

Named by the job it does, because the stack list is the least interesting thing about a pipeline.

Kafka Trip events, keyed by trip_id so one trip's events land on one partition and stay ordered.
Spark Streams to bronze, partitioned by event date, with a trailing window so a late event is reprocessed rather than lost.
dbt Silver to gold: a star schema, then marts, each model tested before anything downstream of it is built.
Airflow One daily DAG with retries, backfill and success checks. The gates run before the load, not after it.
CDC Dimension updates arrive as Debezium-format change streams, so an out-of-order update cannot revert a row.

The events are synthetic. They come from a seeded generator with defects injected on purpose (redeliveries, corrections, broken streams) because a clean dataset cannot exercise a quality gate. What is not circular is the shape of the mistake the pipeline catches.