Optimizer Studio
Watch SGD, Momentum, RMSProp, and Adam descend the same 3D loss surface with the same mini-batch evidence.
A fair optimizer comparison where the dataset, starting parameters, mini-batch IDs, and full-data loss surface are shared. Only the optimizer update rule changes, and every visible path vertex comes from the exact browser-side state.
Step by step
- Fix one dataset, one start point, one mini-batch schedule, and one learning rate.
- Apply plain SGD directly to each mini-batch gradient.
- Add velocity memory with Momentum.
- Track squared-gradient scale with RMSProp.
- Track bias-corrected first and second moments with Adam.
- Replay all four genuine trajectories on the 3D loss surface while a 2D full-data loss chart stays synchronized.
Core formulas
SGD
θₜ₊₁ = θₜ − ηgₜThe current mini-batch gradient is applied directly.
Momentum
vₜ = βvₜ₋₁ + gₜ; θ ← θ − ηvₜVelocity carries useful direction across updates.
RMSProp
sₜ = ρsₜ₋₁ + (1−ρ)gₜ²; θ ← θ − ηgₜ/(√sₜ+ε)Recent squared gradients adapt each coordinate step.
Adam
m̂ₜ, v̂ₜ → θ ← θ − ηm̂ₜ/(√v̂ₜ+ε)Adam combines first-moment direction with second-moment scaling.
When to use Optimizer Studio
- Learning why optimizers can follow different paths even on the same objective.
- Comparing oscillation, memory, and adaptive coordinate scaling.
- Connecting optimizer internals to deep-learning training without changing the existing neural-network engines.