What does the Adam optimizer do, and what problem does it solve over SGD?

How to think about it

Adam is an optimizer, meaning an algorithm that chooses how neural-network parameters change during training. It combines a momentum-like average of recent gradients with a per-parameter scale based on recent squared gradients, so it usually reaches a useful solution faster and with less learning-rate fuss than vanilla SGD. It does not guarantee better final accuracy, and it is not hyperparameter-free.

Why vanilla SGD needs help

A model trains by minimizing a loss, a number measuring how wrong its predictions are. The gradient is the local slope of that loss with respect to each parameter, so moving opposite the gradient should reduce the loss.

Vanilla stochastic gradient descent, or SGD, updates parameters using one global learning rate:

theta_new = theta_old - alpha * g

Here g is the gradient estimated from the current mini-batch, and alpha is the learning rate. A mini-batch is cheaper than processing the entire dataset, but its gradient is noisy. One batch may push a weight right; the next may push it left.

There is another problem. Gradient coordinates can have very different magnitudes. Suppose two parameters receive g = (10, 0.1) and alpha = 0.1. SGD moves them by (-1, -0.01). The first coordinate takes a hundred-times larger step. Lowering the global learning rate protects that coordinate but makes the second one crawl.

The issue is not that SGD is fundamentally wrong. It has no memory of recent gradients and no coordinate-specific adjustment. It asks one knob to control a machine with many differently scaled gears.

What Adam adds

Adam keeps two exponential moving averages. An exponential moving average is a weighted history that gives recent values more influence while gradually forgetting old ones.

m_t = beta1 * m_(t-1) + (1 - beta1) * g_t
v_t = beta2 * v_(t-1) + (1 - beta2) * (g_t ** 2)

mhat_t = m_t / (1 - beta1 ** t)
vhat_t = v_t / (1 - beta2 ** t)

theta_t = theta_(t-1) - alpha * mhat_t / (sqrt(vhat_t) + eps)

m is the first moment, Adam’s smoothed estimate of gradient direction. It behaves like momentum: consistent directions accumulate, while gradients that repeatedly change sign partly cancel.

v is the second raw moment, the smoothed estimate of squared gradient magnitude. It is not exactly statistical variance because it does not subtract the mean. Squaring removes the sign and preserves scale.

The update divides the smoothed direction by the square root of the smoothed magnitude. A coordinate that has repeatedly received large gradients gets a larger denominator and therefore a smaller effective step. A coordinate with small or infrequent gradients gets less scaling. Adam replaces one global step size with a direction from m and a coordinate-wise adjustment from v.

The usual defaults are beta1 = 0.9, beta2 = 0.999, and eps = 1e-8. As a rough intuition, the first average remembers about ten updates and the second about one thousand. These are not hard cutoffs; exponential averages never suddenly forget a value.

eps prevents division by zero. It is numerical protection, not a magic accuracy parameter.

Common misconception: Adam is not learning-rate-free. Its adaptive denominator changes the effective learning rate, but the base alpha still controls the overall size of updates. A bad learning rate can still make Adam diverge or train painfully slowly.

The hats in the equations are bias correction. Both moving averages start at zero, so their early values are biased downward. At step one, with beta1 = 0.9, the raw first moment is only 0.1 * g; dividing by 1 - beta1 restores g. The same logic applies to v. Without correction, the early updates are based on artificially cold estimates.

A numerical example

Take parameters theta = (2.0, -1.0), gradient g = (10.0, 0.1), and an illustrative learning rate of alpha = 0.1.

Vanilla SGD produces:

theta_new = (2.0, -1.0) - 0.1 * (10.0, 0.1)
          = (1.0, -1.01)

For Adam’s first step, bias correction gives:

mhat = (10.0, 0.1)
vhat = (100.0, 0.01)

Ignoring the tiny eps, the normalized direction is:

mhat / sqrt(vhat) = (10 / 10, 0.1 / 0.1) = (1.0, 1.0)

Adam therefore produces approximately theta_new = (1.9, -1.1). The two coordinates take similarly sized steps despite a hundred-fold difference in raw gradient magnitude.

That is the central idea, not a promise that equal-sized steps are always correct. If the large gradient carries genuinely important signal, Adam may damp it too aggressively. Adaptivity is a useful bias, not free intelligence.

The production pattern

For a modern deep-learning model, I would usually start with AdamW rather than coupled Adam when I want weight decay. Weight decay shrinks parameters toward zero.

In coupled Adam, adding an L2 penalty to the gradient means that penalty also passes through Adam’s adaptive denominator. The amount of shrinkage then depends on each parameter’s gradient history. AdamW decouples the two operations:

theta = theta - alpha * adaptive_gradient
        - alpha * weight_decay * theta

A typical PyTorch setup is:

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=1e-3,
    betas=(0.9, 0.999),
    eps=1e-8,
    weight_decay=1e-2,
)

Those values are starting points, not laws. The learning rate remains the most important setting, and a schedule such as warmup followed by decay often matters as much as the optimizer choice.

Save Adam’s optimizer state in checkpoints, not just the model weights. The m and v histories affect the next update. Restoring weights with a freshly initialized optimizer resumes from a different trajectory and can cause a visible training jump.

The senior-level trade-off

Adam’s speed is mainly an optimization advantage, not proof of better generalization. Generalization means performance on unseen data. On many vision and conventional supervised-learning problems, carefully tuned SGD with momentum can match or beat Adam on validation accuracy. Adam may reach low training loss quickly but settle into a solution that transfers less well.

The reverse can also happen. Adam is often attractive when gradients are sparse, parameter scales are uneven, or rapid iteration matters more than squeezing out the final fraction of accuracy.

The comparison must be precise because “SGD” can mean two different things. If the interviewer means vanilla SGD, Adam adds both momentum and adaptive scaling. If they mean SGD with momentum, that optimizer already has the m-like smoothing. Adam’s distinguishing addition is the v-based per-parameter scaling.

I would compare AdamW with momentum SGD using separately tuned learning rates and schedules. Comparing Adam at 1e-3 with an arbitrary SGD rate and declaring a universal winner is not a serious experiment.

There is also a memory cost. Adam stores two extra state values per parameter. For one billion parameters using four-byte state values, those two tensors alone require about 8 GB, before counting parameters, gradients, activations, or sharding overhead. Mixed-precision training often keeps optimizer states in higher precision, so a model that fits in GPU memory may still fail when Adam’s state is allocated. Momentum SGD needs roughly one such state tensor.

A failure you can recognize

A common first symptom is a loss that becomes NaN shortly after training starts. Adam is not automatically unstable. The usual suspects are a learning rate that is too high, exploding gradients, invalid input values, or overflow during mixed-precision arithmetic. Check the data and gradient norms first. Then consider a lower learning rate, correct loss scaling, or gradient clipping when the model’s problem warrants it.

Another symptom is falling training loss while validation performance stalls or worsens. Do not automatically increase beta1. Check the learning-rate schedule, weight decay, and data split, then run a tuned momentum-SGD baseline. If the loss plateaus while gradients remain nonzero, inspect the ratio of update norm to parameter norm: an accumulated v or overly cautious schedule may have made effective updates tiny.

What they’ll ask next

“Is Adam always better than SGD?”

No. Adam is usually easier to get moving and often reaches a good training loss sooner. Tuned momentum SGD can achieve better validation or test performance, particularly in some vision workloads. The objective, batch size, data regime, memory budget, and tuning budget all matter.

“Why do we need bias correction?”

Both moving averages begin at zero, so their early estimates are too small. At step one, the correction divides by 0.1 for beta1 = 0.9 and by 0.001 for beta2 = 0.999, recovering the current gradient and squared gradient under the usual stationary-estimate assumption. It prevents the optimizer from treating its artificial cold start as evidence that gradients are small.

“Why AdamW instead of Adam with weight decay?”

Because Adam’s denominator changes the effect of a penalty when that penalty is added to the gradient. AdamW applies parameter shrinkage separately from adaptive gradient scaling, making the weight-decay setting behave more predictably. I would still tune the value rather than assume 1e-2 is universal.

Say this in the interview

Adam addresses vanilla SGD’s noisy, one-learning-rate-for-every-parameter limitation by combining momentum for a steadier direction with adaptive scaling from squared gradients; it often converges faster, but tuned momentum SGD can generalize better, and Adam still needs learning-rate, schedule, and regularization choices.