Regression Metrics

MSE, RMSE, MAE, R² and the caveats on MAPE: what each punishes, what each hides, and how to choose one from the cost of being wrong.

MSE, RMSE and MAE

▶ lab

Squared error punishes a large miss quadratically and answers in squared units; RMSE restores the units but keeps the outlier sensitivity; MAE is the median-like metric that treats every unit of error the same.

Q · Two delivery-time models have nearly the same RMSE and very different MAE. Which one is better, and what does the difference in the two numbers tell you about how they miss?

R² (Coefficient of Determination)

R² is the fraction of variance the model explains relative to predicting the mean. It can be negative out of sample, it is not comparable across datasets, and a high value can describe a model that is useless for the decision.

Q · A vendor reports an R² of 0.9 for a demand model and your own model scores 0.6 on your data. Is theirs better, and what would it mean if yours went negative on next month's data?

MAPE and Its Caveats

Mean absolute percentage error reads naturally and fails badly: undefined at zero, dominated by small actuals, and asymmetric between over- and under-forecasting. On a demand forecast it blows up on exactly the low-volume items nobody was worried about.

Q · Your demand forecast reports a MAPE that looks poor, but the high-volume products — the ones that matter for revenue — are forecast well. Where is the number coming from, and what is it actually rewarding?

Choosing a Regression Metric

Derive the metric from the cost of being wrong: is a ten-unit miss the same on a hundred-unit item as on a ten-unit item, are large misses catastrophic or merely bad, and is the decision actually a threshold on the forecast — in which case it is classification in disguise.

Q · The team has tried MSE, MAE, R² and MAPE and each ranks the candidate models differently. Which questions about the business decide the metric, and when is the right answer that this is not a regression problem?