Why plain gradient descent keeps failing on real loss surfaces
Plain gradient descent moves every parameter by the same rule: step opposite the gradient, scaled by one global learning rate. That works nicely on a smooth, symmetric bowl. It works much less nicely on the loss surfaces that real neural networks actually produce, which tend to have narrow curved ravines running next to broad flat plateaus. Pick a learning rate large enough to make progress along the flat plateau and the optimizer oscillates wildly across the narrow ravine, sometimes diverging outright. Pick a learning rate small enough to stay stable in the ravine and progress across the plateau slows to a crawl.
The fix that stuck, and the one nearly every modern training run defaults to, is Adam — Adaptive Moment Estimation. Instead of treating every parameter direction identically, Adam keeps two running statistics per parameter: a smoothed estimate of the gradient itself (the first moment, essentially momentum) and a smoothed estimate of the squared gradient (the second moment, which behaves like a per-parameter measure of how large or noisy that direction has been). Dividing one by the square root of the other gives each parameter its own effective step size, automatically compressed in directions that have been consistently steep and left larger in directions that have been quiet.
Two running averages instead of one gradient
The core update, in the same form you'll find in the original paper, is:
m_t = beta1 * m_(t-1) + (1 - beta1) * g_t
v_t = beta2 * v_(t-1) + (1 - beta2) * g_t^2
m_hat_t = m_t / (1 - beta1^t)
v_hat_t = v_t / (1 - beta2^t)
theta_t = theta_(t-1) - alpha * m_hat_t / (sqrt(v_hat_t) + epsilon)
g_t is the gradient of the loss with respect to the parameter at step t. m_t is the exponential moving average of the gradient — this is the momentum term, and it's controlled by the first-moment decay β₁, almost always set around 0.9, meaning roughly 90% of the previous momentum carries forward each step. v_t is the exponential moving average of the squared gradient, controlled by the second-moment decay β₂, almost always set close to 0.999 — much slower-moving than β₁ on purpose, because you want a stable read on "how noisy has this direction been" rather than a twitchy one. The learning rate α then scales the whole normalized step.
Bias correction: fixing the cold start
Both m and v are initialized at zero, which means in the first few steps they're biased toward zero too — especially v, since β₂ is close to 1 and takes a long time to accumulate real signal. Left uncorrected, this makes the very first updates artificially small or, worse, artificially large once you divide by an underestimated v. The m_hat and v_hat terms divide by (1 - beta^t), which is close to zero when t is small and grows toward 1 as t increases, exactly canceling out the cold-start bias. By the time you're a few hundred steps in, 1 - beta1^t and 1 - beta2^t are both essentially 1, and the correction stops mattering.
Walking through the update by hand
Numbers make this concrete faster than more algebra. Take a classic hard case for plain gradient descent: a narrow curved valley defined by
f(x, y) = (1 - x)^2 + 100 * (y - x^2)^2
which has its minimum at (1, 1) with f = 0, and a floor that curves like a banana — exactly the kind of surface where a single global learning rate struggles. Start at (x, y) = (-1, 1), with learning rate α = 0.05, β₁ = 0.9, β₂ = 0.999, ε = 1e-8.
The gradient is:
df/dx = -2(1 - x) - 400*x*(y - x^2)
df/dy = 200*(y - x^2)
At (-1, 1): 1 - x = 2, so the first term is -4. y - x^2 = 1 - 1 = 0, so the second term of df/dx is zero, and df/dy is also zero. So g0 = (-4, 0).
Step 1: m1 = 0.9*0 + 0.1*(-4, 0) = (-0.4, 0). v1 = 0.999*0 + 0.001*(16, 0) = (0.016, 0). Bias-corrected: m_hat1 = -0.4/0.1 = -4, v_hat1 = 0.016/0.001 = 16 for x (y-component stays 0 throughout since its gradient was zero). The x update is -0.05 * (-4)/sqrt(16) = -0.05 * (-1) = +0.05, so x moves from -1.00 to -0.95. That's a modest, controlled step even though the raw gradient magnitude was 4 — the adaptive scaling has already normalized it down to roughly one learning-rate unit, which is exactly the point: Adam's step size in any single direction is bounded near α regardless of how large the raw gradient is.
Run this forward for a few hundred iterations — which is what the simulator does when you set Iterations and press play — and the trajectory curls around the bottom of the valley and settles in. With these hyperparameters, a typical run reports a Final x around 0.97–0.99, Final y around 0.95–0.98, a Final loss f on the order of 1e-3 to 1e-4, a Loss reduction (%) north of 99.9%, and a Verdict of "Converged."
What changes if you push the hyperparameters
Push α up to 0.2 on the same surface and the first few steps overshoot the valley floor and bounce between its walls before settling — still convergent, but visibly noisier in the trajectory plot. Push β₂ down toward 0.9 (closer to β₁) and the second-moment estimate becomes twitchy, reacting to every individual gradient spike instead of smoothing over the ravine's curvature, which reintroduces some of the oscillation Adam is supposed to remove. Push it up past 0.9999 and adaptation becomes so sluggish that early steps behave almost like plain momentum with no per-parameter scaling at all — useful on very smooth losses, actively harmful on ravines like this one.
Where engineers get the hyperparameters wrong
The single most common mistake is treating α as if it needs re-tuning from scratch for every architecture. In practice, 1e-3 to 3e-4 is a reasonable starting point for the vast majority of deep-learning problems, and most of the useful tuning to do first is a learning-rate schedule (warmup then decay) rather than a different fixed value. The second mistake is dropping β₂ to "make training faster" — it does make training louder, not faster, since a noisier second moment gives you less reliable per-parameter scaling, not more useful gradient information. The third, subtler mistake is ignoring epsilon: on parameters where the gradient is consistently near zero, v_hat can get small enough that epsilon (not the gradient) starts dominating the step size, which is usually fine at 1e-8 but can matter on mixed-precision training where you sometimes see it raised to 1e-6 or higher for numerical stability.
Try it yourself
The best way to build intuition for how β₁ and β₂ trade off momentum against per-parameter caution is to watch it happen on a surface where a single learning rate visibly fails. The NovaSolver Adam Optimizer Simulator lets you pick the loss surface, dial in α, β₁ and β₂, set the iteration count, and watch the trajectory converge (or not) toward the minimum, with the final position, loss, and convergence verdict reported live. Try the Adam optimizer simulator here and compare it against the AdaBoost simulator if you want to see a completely different flavor of "adaptive" at work, or the batch normalization tool for the other half of why modern training actually converges in practice.
