What is the difference between Adam and AdamW?

How to think about it

Adam and AdamW share the same Adam machinery: momentum from past gradients and a separate learning-rate adjustment for each parameter. The difference is regularization: classic Adam folds L2 regularization into the gradient, while AdamW applies weight decay directly to the parameters, outside Adam’s adaptive gradient calculation.

Why the distinction exists

Imagine a text classifier whose training loss keeps falling, but whose validation loss has started climbing. You add weight_decay=0.01, expecting the model’s parameters to stay modest and generalize better. That setting does not mean the same thing in Adam and AdamW. The optimizer decides how much each parameter actually shrinks.

Adam maintains two moving averages. The first, m, is an exponentially weighted average of gradients, so it acts like momentum. The second, v, is an exponentially weighted average of squared gradients, so it estimates how large and noisy each coordinate’s gradients have been.

A simplified Adam update is:

m_t = beta1 * m_(t-1) + (1 - beta1) * g_t
v_t = beta2 * v_(t-1) + (1 - beta2) * g_t^2

theta_next = theta - eta * m_hat_t / (sqrt(v_hat_t) + epsilon)

Here, theta means a learned parameter, g is the gradient, eta is the learning rate, and m_hat and v_hat are bias-corrected versions of the two moving averages.

L2 regularization adds a penalty to the loss:

L = L_task + (lambda / 2) * sum(theta_i^2)

Its gradient adds lambda * theta_i to every parameter’s task gradient. Classic Adam therefore sees:

g_i = task_gradient_i + lambda * theta_i

That extra term goes into both m and v. Adam then divides the result by a coordinate-specific quantity based on v. The regularization is no longer a simple, uniform shrinkage of the parameters.

AdamW changes only this part. Its moments use the task gradient alone, and the optimizer performs a separate decay step:

m_t and v_t use task_gradient_t only

theta_next = theta
             - eta * m_hat_t / (sqrt(v_hat_t) + epsilon)
             - eta * lambda * theta

The decay term is equivalent to multiplying the old parameter by 1 - eta * lambda. Every parameter receives the same fractional decay at that step, regardless of its gradient history.

Common misconception: AdamW is not merely a new name for Adam with an L2 penalty. For adaptive optimizers, adding lambda * theta to the gradient and subtracting eta * lambda * theta separately produce different updates.

A concrete two-parameter example

Take two positive parameters:

theta = (2.0, 0.02)
learning_rate = 0.001
weight_decay = 0.1
beta1 = 0.9
beta2 = 0.999
epsilon = 1e-8

Assume this is the first update, the moment estimates start at zero, and the task gradient is zero. We are isolating regularization so the difference is easy to see.

With Adam plus L2 regularization, the gradients are:

(0.1 * 2.0, 0.1 * 0.02) = (0.2, 0.002)

On the first bias-corrected Adam step, the square root of the second moment is approximately the absolute value of each gradient. The adaptive update is therefore about:

0.001 * (0.2 / 0.2, 0.002 / 0.002) = (0.001, 0.001)

The parameters become approximately:

(1.999, 0.019)

The large parameter fell by about 0.05%. The small parameter fell by about 5%. The same L2 coefficient produced dramatically different fractional shrinkage because Adam normalized each coordinate separately.

With AdamW, the task gradient is zero, so the Adam gradient update is zero. The separate decay is:

0.001 * 0.1 * (2.0, 0.02) = (0.0002, 0.000002)

The parameters become:

(1.9998, 0.019998)

Both parameters fell by 0.01%, because AdamW applies a uniform fractional decay. Later updates will include real gradients and nonzero moment estimates, so these exact numbers will change. The mechanism will not: L2 is adaptively scaled; AdamW’s decay is not.

The nuance that earns the senior signal

The textbook statement that “L2 regularization equals weight decay” is true for plain SGD without momentum:

theta_next = theta - eta * (g + lambda * theta)

can be rearranged as:

theta_next = (1 - eta * lambda) * theta - eta * g

There is no per-parameter denominator to distort the penalty. With momentum, even this equivalence needs care, because adding L2 to the gradient also places the penalty inside the momentum buffer. Adam makes the difference more important because it has both momentum and adaptive normalization.

AdamW is often easier to tune and frequently gives better validation performance, especially in transformer training. But it is not a guarantee of better generalization. Too much decay can underfit: training loss may remain high, or useful parameter norms may be suppressed before the model has learned the task. A sensible comparison tunes learning rate and decay on validation data rather than assuming 0.01 is universally correct.

There is also a schedule interaction. AdamW’s fractional decay at step t is eta_t * lambda. If a cosine schedule reduces the learning rate, it also reduces the amount of decay applied per step. With a constant learning rate of 0.001 and decay of 0.1, the per-step factor is 0.9999; after 1,000 steps with no task gradient, the parameter is multiplied by about 0.9048. Under a changing learning rate, the total effect depends on the entire schedule.

In production, decay is usually applied selectively. Matrix weights often receive decay, while biases and scale parameters in LayerNorm or RMSNorm are commonly excluded. Those parameters do not represent model capacity in the same way as a weight matrix, and shrinking them can hurt optimization. This is a convention, not a law; the right parameter groups depend on the architecture and experiment.

Finally, do not infer semantics from the argument name alone. In one optimizer, weight_decay=0.01 may add an L2 term to the gradient. In AdamW, it normally means decoupled parameter decay. Check the optimizer’s documentation and inspect the parameter groups when reproducing a result. If weight_decay is zero and all other settings match, Adam and AdamW have the same Adam core. If you are using plain SGD, AdamW’s particular distinction is irrelevant.

A failure mode to watch for

A common migration mistake is replacing AdamW with Adam while keeping learning_rate=0.001 and weight_decay=0.01. The training loss still decreases, so the smoke test passes. A few epochs later, validation loss is worse than the reference run, and the parameter-norm curve has a noticeably different slope, often with small-norm parameters changing too aggressively.

The root cause is not a broken learning rate. The same numeric decay value is being routed through Adam’s adaptive moments. Fix the optimizer choice first, verify which parameters receive decay, and then retune the decay coefficient rather than copying it blindly.

What they’ll ask next

Is AdamW always better than Adam?
No. AdamW gives more predictable decoupled regularization, and often generalizes better, but the result depends on the dataset, architecture, learning-rate schedule, and decay value. With no decay, the two share the same base Adam update.

Why not just add an L2 term to the loss when using Adam?
You can, but Adam will adaptively scale that penalty just like any other gradient. That is mathematically different from AdamW’s direct parameter shrinkage. Use the loss term when you specifically want a penalty handled as part of the gradient; use AdamW when you want decoupled decay.

Which parameters should receive weight decay?
Usually weight matrices and sometimes embeddings receive it, while biases and normalization scale parameters do not. The important point is to create explicit parameter groups and justify the choice for the model rather than decaying every tensor by default.

Say this in the interview

“Adam and AdamW have the same adaptive-moment update, but Adam mixes L2 regularization into the gradient and therefore scales it per parameter, whereas AdamW decouples weight decay and shrinks the weights directly; that usually makes regularization easier to tune and often improves generalization.”