Dispatch — Last-Mile Delivery Telemetry Pipeline

Dispatch — Last-Mile Delivery Telemetry Pipeline An architecture diagram generated by Archify. Event Generator · synthetic trips + injected defects · Architecture component · Kafka or JSONL Event Generator synthetic trips + injected defects Kafka or JSONL Kafka · trip_events · 3 partitions · Architecture component · at-least-once Kafka trip_events · 3 partitions at-least-once Spark Streaming · foreachBatch + checkpoint · Architecture component Spark Streaming foreachBatch + checkpoint Bronze · raw, partitioned by event date · Architecture component · append-only Bronze raw, partitioned by event date append-only build_silver · conform, dedupe, quarantine · Architecture component · 2 gates build_silver conform, dedupe, quarantine 2 gates Silver · one row per (trip, event) · Architecture component Silver one row per (trip, event) build_gold · sessionize to one row per trip · Architecture component · 7 gates build_gold sessionize to one row per trip 7 gates Star Schema · SQLite local · PostgreSQL in Docker · Architecture component · txn partition replace Star Schema SQLite local · PostgreSQL in Docker txn partition replace dbt Marts · stg_trip → store_daily, promise_error · Architecture component dbt Marts stg_trip → store_daily, promise_error dispatch/ core · contracts · schema_registry · dedupe · sessionize · dispatch/ — no Spark, Kafka or Airflow import · stdlib only dispatch/ core contracts · schema_registry · dedupe · sessionize stdlib only Quality Gates · 8 blocking checks, run before each load · dispatch/ — no Spark, Kafka or Airflow import Quality Gates 8 blocking checks, run before each load Alert Sink · webhook, else task log · dispatch/ — no Spark, Kafka or Airflow import · never raises Alert Sink webhook, else task log never raises Airflow · daily 02:00 · catchup · max_active_runs 1 · Architecture component Airflow daily 02:00 · catchup · max_active_runs 1 produce, keyed by trip_id JSONL when no broker subscribe, maxOffsetsPerTrigger Parquet, dynamic overwrite read partitions + reprocess window rewrite whole partition read events load after gates pass staging → marts + tests normalise + dedupe sessionize enforce before load verdict text on failure schedules ingest → silver → gold → audit dispatch/ — no Spark, Kafka or Airflow import Legend Backend Database Cloud Security Message bus External