MLOps · drift · model registry
The random forest reaches the lowest error on the frozen holdout and ships a 338.5 MB artifact against a declared 50.0 MB budget, so the layer promotes LightGBM instead. An MLOps layer around the used-car price model (car-price-ml): versioned data, MLflow tracking and registry, a drift detector measured for its own false alarms, and promotion that weighs operating cost against error.
Accuracy is only one axis of a production decision. Each candidate is measured on a frozen holdout split alongside its operational cost — artifact size, prediction latency, and batch throughput — so the trade-off is explicit rather than assumed.
| Model | Holdout MAE | Artifact | Predict p50 | Batch throughput |
|---|---|---|---|---|
| Ridge | 15 422 PLN | 0.0 MB | 5.1 ms | 727 554 rows/s |
| LightGBM | 9 278 PLN | 3.3 MB | 12.9 ms | 118 993 rows/s |
| RandomForest | 8 908 PLN | 338.5 MB | 47.1 ms | 148 049 rows/s |
The random forest achieves the lowest error but ships a 338.5 MB artifact and the slowest single prediction; LightGBM reaches within a few percent of that error at roughly one hundredth of the size — the kind of trade-off a registry should record, not hide.
Where these errors come from, and why the sibling's differ. Every MAE above is scored on this project's frozen holdout split, from models trained on its own partition of the corpus. car-price-ml reports the same two model families the other way round, and it is not a contradiction: it scores pooled out-of-fold MAE over 5-fold cross-validation on a de-duplicated corpus. Different question, different answer — the two are reconciled in this repository's README, and the cause of the flip is a question neither repository has answered.
Weekly production snapshots are replayed against six scenarios. The detector is not trusted on faith: it is itself measured for false alarms and detection power, and its population-stability thresholds are calibrated on the project's own data rather than borrowed from the textbook 0.2 rule, which does not hold here.
| Scenario | What changes | MAE change | Alert |
|---|---|---|---|
| stable | no change (control) | −0.9% | no |
| price_shock | prices inflate, features unchanged | +135.2% | yes |
| fuel_mix_shift | more electric and hybrid cars | +2.1% | yes |
| mileage_shift | higher mileage across the week | +24.5% | yes |
| unseen_makes | makes absent from training | +7.5% | yes |
| missing_engine_volume | engine volume arrives empty | +26.1% | yes |
A challenger replaces the champion only when the improvement is shown to be real rather than
fortunate. The decision is made by a paired bootstrap on the frozen holdout: a small nominal gain
is promoted only if its confidence interval clears zero. The winner is registered in MLflow and
served through a stable champion alias, so deployment follows evidence, not a single
lucky evaluation.
mlops-car-price keeps the car-price-ml model alive: reproducible data versioning, MLflow experiment tracking and a model registry, a drift monitor with quantified error rates, bootstrap-based promotion, and an API that serves the current champion. The engineering decisions are recorded as ADRs in the repository.