What is the difference between Gini impurity and entropy as splitting criteria in decision trees?

How to think about it

Gini impurity and entropy are two different ways to score how mixed the labels are in a node of a classification tree: Gini uses squared class probabilities, while entropy uses logarithms and measures average uncertainty in bits. Both reward purer children and usually choose nearly identical splits; Gini is generally a little cheaper, but neither is universally better.

The mechanism the interviewer is probing

A node is the subset of training rows currently sitting at one point in the tree. Suppose a node contains 70 customers who renewed and 30 who cancelled. Its class probabilities are 0.7 and 0.3.

A pure node contains one class only. A mixed node contains several classes. The tree needs a numerical score for that mixedness so it can compare candidate splits.

For a node with class probabilities p_i, the two scores are:

Gini impurity = 1 - sum(p_i^2)

Entropy = -sum(p_i * log2(p_i))

If a class has probability zero, its entropy term is treated as zero.

Gini has a useful probability interpretation. Randomly choose one row from the node, then randomly guess its class using the node’s class proportions. Gini is the probability that the guess is wrong. Equivalently, it is the probability that two independently selected rows have different labels.

Entropy measures average surprise. A common class is unsurprising; a rare class is surprising because -log2(p_i) is larger when p_i is small. Entropy is measured in bits when the logarithm uses base 2.

Both scores are zero for a pure node. For a binary problem, both are largest when the node is evenly split:

Positive-class fraction	Gini	Entropy
0.50	0.500	1.000
0.10	0.180	0.469
0.01	0.0198	0.0808

Do not compare those raw numbers across criteria. Gini’s binary maximum is 0.5; entropy’s is 1 bit. They use different scales.

How a split is scored

The tree does not choose the child with the lowest impurity in isolation. It evaluates the weighted impurity of both children.

For a candidate split with left child size n_left, right child size n_right, and parent size n, the weighted score is:

weighted child impurity =
    (n_left / n) * impurity(left)
  + (n_right / n) * impurity(right)

The tree chooses the split with the largest reduction from the parent’s impurity. With Gini, this is often called Gini decrease. With entropy, the reduction is called information gain, meaning the amount of uncertainty removed by the split.

The weighting matters. A split that creates one tiny pure child is not automatically excellent. If one row becomes a pure child and the other 99 rows remain almost as mixed as before, the gain is tiny. Otherwise, trees would eagerly manufacture a collection of one-row leaves. They are quite capable of that without encouragement.

Concrete example: 100 renewal decisions

Imagine a subscription company with 100 customers at the root node. Forty cancelled and 60 renewed.

The parent has:

Gini    = 1 - 0.4^2 - 0.6^2 = 0.480
Entropy = -0.4 log2(0.4) - 0.6 log2(0.6) = 0.971 bits

Now consider the feature support_contacts, split at three contacts:

• 50 customers had three or more contacts: 30 cancelled and 20 renewed.
• 50 customers had fewer than three contacts: 10 cancelled and 40 renewed.

The first child has a cancellation fraction of 0.6, so its Gini impurity is 0.480 and its entropy is 0.971. The second has a cancellation fraction of 0.2, so its Gini impurity is 0.320 and its entropy is 0.722.

Because the children are equal in size:

Weighted Gini    = 0.5 * 0.480 + 0.5 * 0.320 = 0.400
Gini reduction   = 0.480 - 0.400 = 0.080

Weighted entropy = 0.5 * 0.971 + 0.5 * 0.722 = 0.846
Information gain = 0.971 - 0.846 = 0.125 bits

Now compare a second candidate, late_payment:

• 30 customers paid late: 21 cancelled and nine renewed.
• 70 customers paid on time: 19 cancelled and 51 renewed.

The child Gini impurities are approximately 0.420 and 0.396. Their weighted score is:

0.3 * 0.420 + 0.7 * 0.396 = 0.403

That gives a Gini reduction of about 0.077.

The corresponding child entropies are approximately 0.881 and 0.844. Their weighted entropy is about 0.855, giving information gain of about 0.116 bits.

Both criteria choose the support-contact split in this example:

• Gini: 0.080 beats 0.077.
• Entropy: 0.125 beats 0.116.

The ranking is the same, but the scores are not.

Why the trees usually agree

Gini and entropy have the same broad shape. They both prefer:

• pure nodes over mixed nodes;
• a balanced binary node as the most uncertain case;
• splits that make the children more homogeneous;
• larger improvements spread across many rows rather than accidental purity in a tiny child.

Their mathematical shapes are not identical, though. Entropy’s logarithm gives relatively more weight to a small, nonzero class probability. In the table above, a node with only one percent positives has Gini impurity 0.0198, but entropy 0.0808 bits. After accounting for their different maximum scales, entropy still treats that rare class as more meaningful.

That can matter when two candidate splits distribute a minority class differently. It can also make entropy react more strongly to a noisy rare label. Neither effect guarantees better minority-class recall.

The practical differences are usually small because the best split is often much better than the alternatives. When several thresholds have almost tied scores, however, the criteria can choose different thresholds. That first choice changes the child nodes, which changes every later choice. Two trees that look similar at the root can therefore diverge several levels down.

The senior nuance: what the textbook answer leaves out

Gini is usually a little faster because it needs squaring and addition, while entropy needs logarithms. Do not turn that into a dramatic performance claim. In many implementations, sorting feature values and scanning candidate thresholds costs more than evaluating either formula. The difference is usually a modest training-time consideration, not a reason to sacrifice validation performance.

Both criteria use the same sample-size weighting. Saying that “Gini favors larger partitions” is too loose. A larger child carries more weight under both criteria. The shape of the impurity function can change which candidate wins, but there is no universal rule that Gini prefers one partition size.

Entropy is also not a cure for class imbalance. If fraud is one percent of the data, a leaf that predicts “not fraud” can still look very pure under either criterion. Entropy may respond more to the fraud examples inside a node, but the resulting model still needs appropriate class weights, sampling, decision thresholds, and metrics such as precision-recall curves.

For ordinary binary-split classification trees, I would start with Gini unless there is a reason to prefer entropy’s information-theoretic interpretation. Then I would compare them using the same validation folds and the same complexity controls:

from sklearn.tree import DecisionTreeClassifier

tree_gini = DecisionTreeClassifier(
    criterion="gini",
    max_depth=4,
    min_samples_leaf=20,
    random_state=7,
)

tree_entropy = DecisionTreeClassifier(
    criterion="entropy",
    max_depth=4,
    min_samples_leaf=20,
    random_state=7,
)

The max_depth and min_samples_leaf settings matter more than the criterion in many real projects. They prevent the tree from chasing increasingly tiny, increasingly accidental patterns.

A failure mode you will actually see

Suppose both models reach 100 percent training accuracy, but validation accuracy is 71 percent. The tree diagram contains many leaves with one or two rows, including a branch that effectively says “customer ID 48192 means churn.”

That is overfitting, not evidence that entropy or Gini is wrong. The tree found locally pure splits that memorised noise or an identifier. Limit depth, increase the minimum leaf size, remove leakage, or use cost-complexity pruning. Changing Gini to entropy while leaving the tree unconstrained is usually rearranging the furniture during a fire.

What they will ask next

Does entropy always produce a better tree?

No. It can choose a better split on one dataset and a worse one on another. The differences are data-dependent, and validation performance decides. There is no general accuracy advantage.

Why is Gini commonly the default?

It is simple to compute, usually a little cheaper per candidate split, and tends to produce similar trees. The default is an engineering convention, not a mathematical proof that Gini is superior.

What changes when the classes are severely imbalanced?

Neither criterion solves the problem. Use class weights or resampling when appropriate, choose a business-relevant decision threshold, and evaluate minority-class performance with metrics suited to the problem. A lower impurity score is not the same thing as useful fraud detection or fair medical triage.

Say this in the interview

“Gini uses squared class probabilities, entropy uses logarithms to measure uncertainty, and both choose the split with the greatest weighted reduction in child impurity; they usually agree, Gini is slightly cheaper, and regularization and validation matter more than the choice between them.”