Explain how gradient boosting fits residuals. What role does the learning rate play?

How to think about it

Gradient boosting fits each new tree to the current model’s mistakes, or more precisely to the negative gradient of the loss. The learning rate, often written as eta, shrinks that tree’s contribution, so the ensemble corrects itself in smaller steps instead of allowing one tree to swing the prediction too far.

Why residuals appear

A gradient-boosted model is an additive model, meaning it starts with a simple prediction and adds one correction at a time:

F_m(x) = F_{m-1}(x) + eta * h_m(x)

Here, F_{m-1} is the ensemble so far, h_m is the new tree, and m is the boosting round.

Suppose we are predicting house prices. The first model might predict the same value for every house. For squared-error regression, that value is the mean house price because the mean is the constant that minimises the sum of squared errors.

After that, the model examines how wrong each prediction is. A residual is the actual value minus the current prediction. If a house is worth $300,000 and the model predicts $250,000, its residual is $50,000. The next tree learns patterns in those residuals from the original features, such as size, location, and number of bedrooms.

The next tree is not trained to predict the house price again. It is trained to predict the correction needed by the current ensemble.

The full algorithm is:

1. Start with an initial prediction F_0.
2. Calculate one target for every training row: the negative gradient of the loss.
3. Fit a tree h_m from the features to those targets.
4. Add a shrunken version of that tree to the ensemble.
5. Recalculate the targets using the updated predictions.
6. Repeat.

The important phrase is using the updated predictions. The residuals change after every tree. Gradient boosting does not keep fitting every tree to the original error.

For squared error, using the loss 1/2 * (y - F(x))^2, the negative derivative with respect to the prediction is exactly y - F(x). That is the ordinary residual.

For other losses, the target is called a pseudo-residual, meaning a gradient-based correction that behaves like a residual but is not necessarily an actual difference between two values.

A concrete example

Imagine four houses. Prices are measured in thousands of dollars.

House	Actual price	Initial prediction	First residual
A, 1,000 sq ft	180	240	-60
B, 1,500 sq ft	220	240	-20
C, 2,500 sq ft	260	240	20
D, 3,000 sq ft	300	240	60

The initial prediction is 240, the mean of the four prices.

Suppose the first tree is a stump that splits at 2,000 square feet. In the smaller-house leaf, the mean residual is:

(-60 + -20) / 2 = -40

In the larger-house leaf, it is:

(20 + 60) / 2 = 40

So the tree says:

• Smaller houses need a correction of -40.
• Larger houses need a correction of +40.

With a learning rate of 0.5, the ensemble applies only half of each correction:

House group	Old prediction	Tree output	Applied correction	New prediction
Smaller houses	240	-40	-20	220
Larger houses	240	40	20	260

The new residuals are now:

• House A: 180 - 220 = -40
• House B: 220 - 220 = 0
• House C: 260 - 260 = 0
• House D: 300 - 260 = 40

The second tree therefore sees residuals of -40, 0, 0, 40. With the same split, its leaf predictions are -20 for smaller houses and 20 for larger houses. At a learning rate of 0.5, it moves the predictions to 210 and 270.

The third tree sees a smaller remaining pattern. The model is walking toward the correct group averages rather than jumping there in one move.

With a learning rate of 1, the first tree would move the smaller-house prediction from 240 to 200 and the larger-house prediction from 240 to 280 immediately. That is correct for this simple split, but real data contains noise and more complicated interactions. A full-strength tree can make an aggressive correction based on a pattern that does not hold outside the training sample.

A tree’s leaf output is the mean residual for squared-error loss because that mean is the best constant prediction for the residuals in that leaf. For other losses, implementations may choose the leaf value by directly minimising the original loss, often using a line search. That is one reason “gradient boosting fits residuals” is a useful intuition but not the complete mathematical description.

What the learning rate changes

The learning rate controls how much of each tree the ensemble accepts. It is also called shrinkage, meaning that every tree’s output is scaled down before being added.

A smaller learning rate does three things:

• It makes each update more conservative.
• It usually requires more boosting rounds to reach the same training loss.
• It often improves generalisation because an individual tree has less power to encode noise.

The number of boosting rounds is usually controlled by n_estimators. The learning rate and n_estimators must be tuned together. A model with a learning rate of 0.03 and 2,000 trees is not directly comparable to one with a learning rate of 0.1 and 200 trees. The first is allowed many smaller corrections; the second makes fewer, larger corrections.

A useful starting intuition is that reducing the learning rate requires increasing the tree budget, sometimes roughly in inverse proportion. It is not an exact conversion. Changing the learning rate changes the residuals seen by later trees, so the later trees are not simply copies of the earlier ones.

A small learning rate is not automatically better. At a fixed number of trees, it can leave both training and validation performance poor. That is underfitting, not healthy regularisation. It also increases model size and prediction cost if the model needs thousands of trees.

In practice, I would tune a small grid of learning rates and let validation performance determine how many trees are useful:

from sklearn.ensemble import GradientBoostingRegressor

model = GradientBoostingRegressor(
    n_estimators=2000,      # an upper bound, not necessarily the final count
    learning_rate=0.03,
    max_depth=2,            # limits the complexity of each correction
    subsample=0.8,          # each tree sees 80 percent of the rows
    n_iter_no_change=30,    # stop when validation progress stalls
    validation_fraction=0.1,
    tol=1e-4,
    random_state=42,
)

model.fit(X_train, y_train)

subsample=0.8 introduces stochastic gradient boosting: each tree uses a random 80 percent sample of the training rows. This can reduce overfitting by making the sequence less dependent on every noisy row, but it also adds randomness and may require more trees. The trees remain sequential; this is not the same as a random forest, where trees are trained independently and then averaged.

For time-dependent data, a random validation fraction can be inappropriate because it may put future information into the validation design. I would use a time-ordered validation split instead.

Residuals are not always ordinary errors

The phrase “fit the residuals” is exact for squared-error regression, but it becomes shorthand for classification and other objectives.

For binary classification with the usual logistic loss, the ensemble score F(x) can be interpreted as a log-odds value. Convert it to a probability p with the sigmoid function. The negative gradient is y - p.

For a positive example with y = 1 and p = 0.8, the pseudo-residual is 0.2. The model is already moving in the right direction, so the required correction is modest.

For a negative example with y = 0 and p = 0.8, the pseudo-residual is -0.8. The model is confidently wrong, so the next tree gets a much stronger signal.

That is different from fitting a binary target of zero or one, and different again from fitting only the misclassified examples. Correctly classified but poorly calibrated examples still contribute gradients.

For losses such as Huber loss, the correction also depends on the loss’s gradient. Absolute error is not differentiable at exactly zero, but boosting can use a subgradient, which is a valid slope choice at that point.

The failure mode to watch for

The classic symptom is a falling training loss paired with a validation loss that first improves and then rises. For example, training RMSE might keep dropping from 18 to 9 while validation RMSE reaches its best value of 16 after 640 trees and then climbs to 18 by tree 1,500.

Late trees are then fitting quirks of the training data rather than reusable structure. A smaller learning rate can delay this point, but it cannot make unlimited boosting safe. A deep tree, noisy features, duplicated records, or a leaked feature can still produce an overfit model.

Early stopping, a held-out validation set, and a sensible tree depth are the usual controls. The best model is the one at the validation minimum, not the one with the lowest training loss.

What they’ll ask next

Are the residuals always y - prediction?
No. That is exact for squared-error regression, assuming the usual half-squared-error convention. In general, the target is the negative gradient of the chosen loss. For logistic classification, it is based on y - p; for other objectives, it follows that objective’s gradient.

Why not use one very deep tree instead?
A deep tree can fit complex interactions in one step, but it can also memorise noise. Boosting uses smaller trees as controlled corrections. Depth controls the complexity of each correction, while the learning rate controls its magnitude. Both affect overfitting.

Does a smaller learning rate always produce a better model?
No. It often helps when paired with enough trees, but at a fixed tree budget it can underfit. It also increases training time, memory, and prediction latency when many more trees are needed. I would compare learning-rate and tree-count combinations on validation data, with the production cost included.

Say this in the interview: Gradient boosting repeatedly fits each new tree to the current negative-gradient errors and adds only a learning-rate-scaled correction, so a smaller learning rate usually generalises better but requires more trees and careful early stopping.