What is pruning in decision trees and when would you use pre-pruning versus post-pruning?

How to think about it

Pruning is regularization for a decision tree: it prevents branches that memorize training noise from surviving into the final model. Pre-pruning stops the tree while it is growing; post-pruning grows a larger tree first and then collapses weak subtrees. I use pre-pruning when training cost, latency, or a quick baseline matters, and post-pruning when I need one compact, interpretable tree and can afford validation.

Why pruning works

A decision tree repeatedly splits the feature space. At each node, it chooses the split that gives the largest immediate reduction in impurity, where impurity measures how mixed the target values are. For a binary classification node, Gini impurity is 1 - p² - (1 - p)², where p is the fraction of examples in one class. A node containing only one class has impurity zero.

This greedy choice is useful, but it has no natural reason to stop at the point where the pattern becomes meaningful. If the algorithm is allowed to continue, it can create a leaf for a handful of unusual customers, one mislabeled image, or one accidental combination of features. The tree then has very low training error but high variance: small changes in the training data produce a different tree.

Pruning trades a little bias, meaning some training patterns are deliberately ignored, for lower variance, meaning the model is less sensitive to accidental details. That usually improves generalisation, which means performance on new data rather than on the examples used for training.

The underlying idea is covered in more detail in decision trees, but the interview distinction is about when the restriction is applied.

A concrete example

Imagine a loan-default model trained on 10,000 applications: 7,000 for training and 3,000 held out until the end. The features include income, debt-to-income ratio, number of late payments, and account age.

Suppose an unrestricted tree grows to 63 leaves and depth 16. It achieves 99.4 percent training accuracy but only 84.8 percent mean five-fold cross-validation accuracy. The small leaves are learning quirks such as “this exact combination of an old account and three late payments happened to default in this sample.”

After cost-complexity pruning, suppose cross-validation selects an 11-leaf tree. Its training accuracy falls to 91.8 percent, but cross-validation accuracy rises to 87.2 percent. That drop in training accuracy is not a failure. It is evidence that the tree stopped trying to explain every training example individually.

Those numbers are illustrative, not a promise that pruning always produces the same improvement. The point is the shape of the result: training performance gets worse while held-out performance gets better.

Pre-pruning: stop growth early

Pre-pruning applies a rule before creating a split, or while the tree is being built. Common controls include:

Parameter	What it limits
max_depth	The longest root-to-leaf path
min_samples_split	The number of samples needed before a node may split
min_samples_leaf	The minimum number of samples allowed in each resulting leaf
max_leaf_nodes	The total number of terminal leaves
min_impurity_decrease	The minimum impurity improvement required for a split

For example, min_samples_leaf=20 rejects any split that would create a leaf containing fewer than 20 training examples. This is often more useful than min_samples_split alone. A node might contain 100 examples and satisfy min_samples_split=40, yet a proposed split could still create one child containing a single example. min_samples_leaf prevents that.

The main advantage is cost. The algorithm never builds rejected subtrees, so the tree uses less memory and trains faster. A depth limit also gives a clear upper bound on prediction work: a prediction follows at most that many decisions.

The main weakness is that pre-pruning is locally short-sighted. A split with little immediate benefit can enable an excellent split one level later.

Consider the XOR pattern:

x1	x2	target
0	0	0
0	1	1
1	0	1
1	1	0

Splitting on x1 first leaves one zero and one one in each child. Splitting on x2 has the same problem. The immediate impurity reduction is zero, even though two levels of splits represent the pattern perfectly. A positive min_impurity_decrease, or a depth limit of one, stops too early.

Common mistake: saying that pre-pruning simply “prevents overfitting.” It can also create underfitting by blocking a useful interaction before the tree has a chance to express it.

Post-pruning: grow, then simplify

Post-pruning first builds a larger tree and then evaluates whether each subtree earns its complexity. A subtree can be replaced by one leaf. In classification, that leaf usually predicts the majority class or class probabilities; in regression, it usually predicts the mean target value.

One approach is reduced-error pruning. Keep a validation set, replace a subtree with a leaf, and retain the replacement if validation loss does not get worse. This directly measures the desired outcome, but it requires a validation split or cross-validation and therefore leaves fewer examples for fitting.

Scikit-learn provides minimal cost-complexity pruning. It scores a tree with:

R_alpha(T) = R(T) + alpha × |leaves(T)|

Here, R(T) is the tree’s total weighted leaf impurity, |leaves(T)| is the number of leaves, and alpha is the price charged for each leaf. With alpha set to zero, there is no additional complexity penalty. As alpha increases, larger and weaker subtrees become too expensive and are collapsed.

For example, suppose tree A has eight leaves and impurity risk 0.18, while tree B has three leaves and risk 0.23.

• At alpha=0.02, tree A scores 0.18 + 0.02 × 8 = 0.34.
• Tree B scores 0.23 + 0.02 × 3 = 0.29, so the smaller tree wins.
• At alpha=0.005, tree A scores 0.22 and tree B scores 0.245, so the larger tree wins.

The best alpha is not universal. Its scale depends on the data, sample weights, target, and impurity measure.

A typical scikit-learn workflow is:

from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.tree import DecisionTreeClassifier

probe = DecisionTreeClassifier(random_state=0)
path = probe.cost_complexity_pruning_path(X_train, y_train)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

search = GridSearchCV(
    DecisionTreeClassifier(random_state=0),
    param_grid={"ccp_alpha": path.ccp_alphas},
    scoring="roc_auc",
    cv=cv,
    n_jobs=-1,
)

search.fit(X_train, y_train)

final_tree = search.best_estimator_
best_alpha = search.best_params_["ccp_alpha"]

cost_complexity_pruning_path supplies the effective alpha values at which the tree changes shape. The final value may produce a stump, meaning a tree with one leaf, and should be allowed as a candidate rather than rejected automatically. Cross-validation chooses the alpha using only X_train and y_train; the untouched test set should be used once for the final estimate.

Choose the scoring metric deliberately. If defaults are rare, accuracy may reward a heavily pruned tree that predicts “no default” almost everywhere. A model can move from 95 percent to 96 percent accuracy while default recall falls from 61 percent to 38 percent. Use a metric tied to the decision, such as recall, precision, average precision, ROC AUC, or a cost-weighted score.

Which should you use?

Pre-pruning is a good choice when the tree must be cheap to train, bounded in depth, or deployed under strict latency and memory limits. It is also a sensible first baseline.

Post-pruning is usually preferable for a standalone tree whose size and explanation matter. It lets the tree discover interactions before asking whether the resulting branch is worth keeping. The price is extra training and validation work because many candidate trees must be evaluated.

The two techniques can be combined. For example, a reasonable min_samples_leaf can prevent pathological one-example leaves, while cost-complexity pruning selects the final overall size. Do not set those limits so aggressively that the larger tree can no longer discover important structure.

For random forests, deep individual trees are often intentional. A random forest reduces variance by averaging many decorrelated trees, so pruning every tree can add bias without providing the same benefit it provides to one standalone tree. For gradient boosting, shallow trees are already part of the weak-learner design; depth, learning rate, and number of boosting rounds are normally tuned together. Pruning inside either ensemble can be tested, but it should not be assumed to help.

Failure modes to watch for

If a pre-pruned tree has depth two, training accuracy is low, and cross-validation accuracy is similarly low, the tree is probably underfitting. A setting such as min_samples_leaf=500 on 7,000 training rows may be preventing useful regions from forming. Relax the constraint and retune it rather than assuming the data has no signal.

If accuracy improves after pruning but minority-class recall collapses, the pruning objective is wrong for the application. Inspect per-class metrics and use class-aware scoring.

If the best cross-validated alpha is zero, that is also a result: the data provides no evidence that pruning helps under the chosen metric. In that case, adding complexity penalty merely because “pruning is supposed to help” would be cargo cult.

What they’ll ask next

Does pruning always improve test accuracy?

No. It reduces variance at the cost of bias. If the original tree is already too shallow, or the dataset is large and clean, pruning can make performance worse. The choice must be validated against held-out data.

Why not just set max_depth?

max_depth is a blunt global limit. It may block a useful interaction simply because that interaction occurs several levels down one path. Post-pruning evaluates the value of the resulting subtree, not just its depth. In practice, I would tune max_depth, leaf-size constraints, and ccp_alpha with cross-validation rather than treating any one setting as universally correct.

How do you choose the final alpha?

Generate candidates from the training data, evaluate them with cross-validation using the real business metric, and keep the test set untouched. If several values perform within normal validation noise, I prefer the simpler tree because it is easier to explain and usually more stable.

Say this in the interview: “Pre-pruning stops a tree during growth and is cheaper, while post-pruning grows a larger tree and then removes subtrees using validation or a complexity penalty; I choose between them based on data size, compute limits, interpretability, and the validation metric.”