Arrival · point-in-time feature store
generated 2026-10-09 18:21 UTC from 19,312 trips · 01 Sep to 12 Sep · every figure computed at build time
Lower is better. The flat promise is what the business does today, so it is the only baseline an improvement means anything against.
The gap is the finding. Filtering a feature window on when a trip was assigned rather than when it finished buys 0.92 minutes of MAE, more than the honest model's entire lift over the baseline, and none of it survives production, because at prediction time those trips had not finished and one of them is the trip being predicted. Dropping the window altogether gaps only 0.09 minutes: small enough to dismiss as noise, wrong for exactly the same reason, and the shape a GROUP BY written without a date filter actually takes.
Every estimate doubled from 09:00. Trips take 35 minutes, so a trip's true duration is only known 35 minutes after it was predicted. This is what the accuracy dashboard showed. Click or press Enter on a column heading to sort.
| Time | Trips scored | MAE (healthy) | MAE (broken) | Visible? | Labels pending |
|---|---|---|---|---|---|
| 09:20 | 46 | 0.500 | 0.500 | identical | 154 |
| 09:40 | 66 | 0.500 | 3.636 | diverged | 134 |
| 10:00 | 86 | 0.500 | 10.930 | diverged | 114 |
Twenty minutes into the outage the dashboard reads the same MAE as a model that never broke, because every trip it could score was predicted before 09:00. Accuracy is structurally behind by the label delay. Prediction drift is not. Predictions exist the moment they are served, and the broken model's distribution has already moved while its accuracy has not.
Latency, reported as percentiles rather than a mean.
The mean is 106 ms and would clear any alarm anyone would set. One request in twenty takes 2.0 seconds. Reporting a mean latency is how a service stays slow without anybody finding out.
A treatment that genuinely is 2% better, 0.16 min off an 8 min MAE, over 800 trips.
The measured difference sits inside what this experiment could resolve, so it is noise with a decimal point. Without the third number the second one goes into a deck as a win either way.
Trips come from Dispatch's gold table, where
assigned_at is the instant a prediction was due and
tat_minutes only exists once the trip ends. Features are
declared once and retrieved two ways: point-in-time for training, by
entity for serving, and a test replays a day asserting the two paths
return identical values for every trip.
Architecture diagram → · What Dispatch hands Arrival →
Trips are synthetic where Dispatch's warehouse is absent, so treat the absolute minutes as a demonstration. What is not circular is the ordering: a leak flatters a model, the flattery does not survive serving, and an accuracy dashboard cannot see an outage until its labels arrive.