Model Evaluation Studio
Move the decision threshold and watch individual held-out scores become TP, FP, TN, FN, metrics, ROC/PR operating points, and calibration evidence.
A lightweight held-out-score studio for binary classification evaluation. The learner traces one unchanged set of prediction scores through thresholding, a confusion matrix, precision/recall/F1, ROC and precision–recall curves, calibration bins, ROC AUC, and Brier score without retraining a model.
Step by step
- Start with continuous held-out prediction scores and true binary labels.
- Choose a decision threshold and convert each score into a predicted class.
- Place every held-out sample into TP, FP, TN, or FN.
- Calculate accuracy, precision, recall, specificity, false-positive rate, and F1 from those counts.
- Sweep the same scores across all unique thresholds to construct ROC and precision–recall curves.
- Compare predicted score with observed positive frequency in calibration bins and inspect Brier score.
Core formulas
Precision
Precision = TP / (TP + FP)Among predicted positives, the fraction that are truly positive.
Recall / TPR
Recall = TP / (TP + FN)Among actual positives, the fraction found by the threshold.
Brier score
BS = (1/n) Σ (pᵢ − yᵢ)²A probability score that penalizes squared distance between predicted probability and the binary outcome.
When to use Model Evaluation Studio
- Learning why one classification threshold cannot optimize every metric at once.
- Understanding confusion matrices, ROC, precision–recall, and calibration from individual predictions.
- Seeing which metrics change with the threshold and which depend only on the underlying scores.