Model Evaluation Foundations · Intermediate

Bias–Variance Studio

Move model complexity and watch fit, train error, validation error, bias², and variance change together.

A synthetic known-signal experiment that keeps training and validation evidence separate. Polynomial complexity is replayable, while repeated deterministic training sets estimate how much prediction error comes from systematic bias, model variance, and irreducible noise.

Step by step

  1. Generate noisy samples from one known nonlinear signal.
  2. Fit a polynomial using training points only.
  3. Measure train and held-out validation MSE at the same degree.
  4. Sweep degree from simple to complex and watch the validation curve.
  5. Refit across repeated deterministic training sets.
  6. Estimate average bias² and prediction variance against the known signal.

Core formulas

Bias

Bias(x) = E[ŷ(x)] − f(x)

Bias measures systematic deviation of the average learned prediction from the true signal.

Variance

Var(ŷ(x)) = E[(ŷ(x) − E[ŷ(x)])²]

Variance measures how much predictions change across training sets.

Error decomposition

Expected error ≈ Bias² + Variance + Noise

For this known synthetic process the three terms can be visualized separately.

When to use Bias–Variance Studio

  • Understanding underfitting and overfitting beyond a single training score.
  • Learning why validation error can rise while training error continues to fall.
  • Seeing why more data or simpler models can reduce variance in some settings.

Primary references

Google ML Crash Course — Overfitting

scikit-learn — Underfitting vs. Overfitting

scikit-learn — Validation and learning curves