AI Foundations · Intermediate

Gradient Sampling Lab

Compare full-batch, mini-batch, and one-example stochastic gradients on one genuine loss surface.

A standalone two-parameter regression lab for the gradient-source question: how does using all examples, a subset, or one example change the update path? The full-data objective stays fixed while the gradient evidence changes.

Step by step

  1. Start from one deterministic two-feature regression dataset and one shared parameter point.
  2. Use the full dataset to calculate a stable batch gradient.
  3. Use a deterministic subset to calculate a mini-batch gradient.
  4. Use one example to calculate a stochastic gradient.
  5. Compare all three genuine parameter trajectories on the same full-data loss surface.
  6. Inspect which sample IDs produced the current update, then continue to Optimizer Studio for SGD, Momentum, RMSProp, and Adam.

Core formulas

Batch gradient

g_B = (1/|B|) Σᵢ∈B ∇ℓᵢ(θ)

The batch B decides which examples contribute to the current gradient estimate.

Full-batch update

B = all training examples

Using every example produces the least noisy gradient estimate for this fixed dataset.

Stochastic update

|B| = 1

One-example gradients update frequently but can be noisy relative to the full-data objective.

When to use Gradient Sampling Lab

  • Understanding the difference between batch, mini-batch, and stochastic gradient estimates.
  • Seeing why smaller batches create noisier optimization paths.
  • Preparing for the separate Optimizer Studio.

Primary references

Google ML Crash Course — Learning rate and batch size

Google ML Glossary — Batch and stochastic gradient descent