Practice is not the exam
A model should be evaluated on data it did not use to fit its parameters.
The idea
Practice questions teach you; unseen questions tell you whether you actually learned instead of memorized.
Training data is used to fit the model. Held-out test data estimates how the fitted model behaves on unseen samples. Leakage makes that estimate unreliable.
How it works
- Partition data using a strategy that matches how the system will encounter future examples.
- Fit preprocessing only on training data, then apply the fitted transformations to validation/test data.
- Use validation or cross-validation for model and hyperparameter selection.
- Evaluate once on a protected test set after the development decisions are fixed.
Generalization gap
A positive and growing gap can be a warning sign of overfitting. The absolute values still depend on the task, metric, sample size, and whether the splits represent the same deployment distribution.
gap = L_validation − L_train
Check your understanding
Why keep a test set separate during fitting?
- To make the dataset larger
- To estimate behavior on unseen data
- To increase the number of features
- To remove all noise
Answer and explanation
A held-out test set gives evidence about generalization because those samples were not used to fit the parameters.
A common misconception
A test set is not automatically trustworthy just because it is named ‘test’. It must be protected from the decisions you make during development.
ML Basics
Learn the learning loop before diving into individual models: data, splits, regression, and probabilities.