Measure the right kind of error
Different tasks and mistakes need different metrics. A useful metric must match the question you care about.
The idea
A single score is only useful if it measures the mistake you actually care about.
Classification can use confusion-matrix-derived metrics such as precision, recall, or F1; regression can use MAE, MSE/RMSE, R², and related measures depending on the objective.
How it works
- Choose held-out predictions from a realistic evaluation split.
- Compute task-appropriate metrics.
- For score-based classifiers, inspect behavior across thresholds and choose an operating point using validation data and costs.
- Check calibration, subgroup performance, uncertainty, and distribution shift when those matter to deployment.
Precision and recall
Precision asks how many predicted positives were truly positive; recall asks how many actual positives were found.
precision = TP/(TP+FP), recall = TP/(TP+FN)
Check your understanding
Why might accuracy alone be misleading on an imbalanced classification dataset?
- A majority-class predictor can look strong while missing the minority class
- Accuracy cannot be calculated
- It changes the training labels
- It always equals recall
Answer and explanation
When one class dominates, predicting that class often can produce high accuracy while performing poorly on the minority class.
A common misconception
A high AUC or accuracy does not guarantee useful calibrated probabilities, fair subgroup behavior, or good performance at your chosen operating threshold.
Data & AI Foundations
Understand distributions, projections, evaluation, and explanations so model outputs are easier to reason about.