Machine Learning Engineer
mediummachine-learning-engineer-online-vs-offline-evaluation
Online vs offline evaluation: how do you decide if a model is better?
Answer
Offline metrics (AUC, RMSE) measure historical performance, but online metrics measure real product impact.
Best practice:
- Start with offline gating
- Run shadow/canary
- Validate with A/B tests when possible
Watch for distribution shift and metric mismatch: improving offline scores doesn’t guarantee better user outcomes.
Related Topics
EvaluationExperimentationMLOps