Model Interpretability — SHAP vs LIME

The SHAP lesson covered the mechanics of one method in depth. This lesson zooms out: it places SHAP next to LIME — its most common rival for single-prediction explanations — and next to the rest of the toolkit you’ll actually use. The goal isn’t to crown a winner. It’s to know which question each method answers, and to stop misreading the output.

The one distinction that organizes everything: global vs local

Every post-hoc method answers one of two questions:

• Global — How does the model behave on average, across the whole dataset? Which features does it rely on most? What’s the average shape of a feature’s effect? Examples: permutation importance, partial dependence (PDP).
• Local — Why did the model produce this prediction for this one instance? Which features pushed this specific output up or down? Examples: LIME, a single SHAP explanation, ICE curves.

The neat part: SHAP lives in both columns. It is fundamentally a local method — one Shapley attribution per feature, per prediction — but those per-instance values stack into global views (mean absolute SHAP for importance, dependence/beeswarm plots for effect shape). That dual nature is a big part of why it’s so widely used.

LIME — fit a simple model that’s right near here

LIME (Local Interpretable Model-agnostic Explanations; Ribeiro, Singh, and Guestrin, “Why Should I Trust You?”, KDD 2016) has a gloriously simple idea. You can’t make a complex model simple everywhere — but you can approximate it with a straight line in the tiny neighborhood around one prediction. The recipe is four steps:

1. Perturb — sample synthetic points around the instance you care about.
2. Predict — ask the black-box model for its output on each perturbed point.
3. Weight — weight those samples by proximity to the original instance (a kernel pi_x — closer points matter more).
4. Fit — train a simple, sparse linear surrogate on this weighted local dataset. Its signed coefficients are the explanation.

Formally, LIME solves an objective with two pieces — a proximity-weighted fidelity loss and a complexity penalty:

LIME’s strengths are real: it is model-agnostic (it only queries predictions — it never looks inside f) and fast. Its weaknesses are equally real:

• It aims only for local fidelity. The surrogate is a decent approximation right next to your instance and says nothing about the model elsewhere — the paper explicitly trades global faithfulness for local faithfulness.
• It is unstable. Because steps 1–2 use random sampling, re-running with a different seed can produce a different explanation for the same prediction, and two nearby points can get very different explanations. It’s also sensitive to the neighborhood kernel width, for which there’s no principled default. (This instability has spawned a stream of stabilization variants — S-LIME, DLIME, SLICE.)

SHAP — distribute the exact prediction, fairly

SHAP (SHapley Additive exPlanations; Lundberg and Lee, A Unified Approach to Interpreting Model Predictions, NIPS 2017) starts from a different place: cooperative game theory. Treat each feature as a player in a game whose payout is the prediction, and ask for each feature’s Shapley value — its fair average marginal contribution across all coalitions of features. The defining property is additivity (also called local accuracy):

f(x) = base_value + Σ_j  shap_value_j

where base_value is E[f(x)], the model’s average output. The contributions sum exactly to (prediction − base value). That sum-to-the-prediction guarantee is the thing LIME’s surrogate does not give you. SHAP is pinned down by three axioms — local accuracy (sums to the output), missingness (an absent feature gets zero), and consistency (if a feature’s marginal contribution never decreases, its SHAP value can’t decrease) — and consistency is exactly what LIME and naive gain-based tree importance lack.

There are two SHAP estimators you’ll meet:

KernelSHAP is the bridge between the two worlds: it estimates Shapley values by solving a specially-weighted linear regression over sampled feature coalitions — essentially LIME’s local-surrogate framing, but with the SHAP kernel that makes the result Shapley values. TreeSHAP is a different, tree-specific algorithm that is exact, not a sampler.

See it: the same prediction, two explanations

Here is the whole comparison in one place. One loan applicant, one black-box output of 0.74 default risk. Toggle between SHAP (a waterfall that starts at the base value and sums exactly to the prediction) and LIME (signed surrogate weights that approximate the model near this point). Then flip the scope to global to swap the per-instance attribution for permutation-importance bars — a different question entirely.

The aha the widget is built around: LIME approximates the model locally with a surrogate; SHAP fairly distributes the exact prediction. Same question (“why this prediction?”), different guarantees. Permutation importance answers a third question — not “why this row?” but “how much does the model lean on each feature overall?” — which is why its ranking can legitimately differ from a single SHAP explanation.

The rest of the global toolkit

Two more staples round out the map. Both are global, both are built into scikit-learn’s inspection module.

• Permutation importance (Breiman, 2001). Measure a baseline score, randomly shuffle one feature’s column to break its link to the target, re-score, and take the drop in performance as that feature’s importance. Bigger drop = more reliance. Crucially it’s computed on held-out data and — unlike default impurity/gain importance — is not biased toward high-cardinality features.
• Partial Dependence Plot (PDP) (Friedman, 2001). Vary one feature across its range, average the prediction over all other features, and plot the result. It shows the average shape of that feature’s effect — is risk rising, falling, flat, or U-shaped as the feature grows?
• Individual Conditional Expectation (ICE) (Goldstein et al., 2015) disaggregates the PDP: one line per instance instead of the average. The PDP is literally the mean of all ICE lines. ICE exposes heterogeneity — if half your population rises and half falls, the PDP shows a flat line that’s true of nobody, and only ICE reveals it.

How to actually READ each output

This is where people go wrong. Each method’s output answers a specific question — read it as that, nothing more:

The single most common misread: treating an importance number (permutation or mean |SHAP|) as if it showed direction. It doesn’t. Importance only ranks reliance. For direction and shape you need signed SHAP values, a PDP, or ICE.

The gotchas — all from the methods’ own literature

The rest, briefly:

• Correlated features split credit. When two features are correlated, the attribution (and PDP, and permutation importance) gets divided between them — or evaluated at unrealistic feature combinations — so a single feature’s number can mislead. The SHAP-in-production field guide walks through a real case where credit went to the wrong correlated cousin.
• PDP and ICE assume independence. They vary one feature while holding others fixed; under strong correlation they evaluate points that never occur in reality.
• LIME is unstable and kernel-width-sensitive (above) — always check whether the explanation holds across seeds before you trust it.
• Default tree feature_importances_ (gain/impurity) is biased and inconsistent — it favors high-cardinality features and was a core motivation for both permutation importance and TreeSHAP.

When to reach for which

The practical default: tree ensemble? Use TreeSHAP — it’s fast, exact, consistent, and does local and global at once. Arbitrary black box and you just need a fast single-prediction sketch? LIME — but verify it across seeds. Want principled Shapley values regardless of model type and can eat the runtime? KernelSHAP. For a cheap global ranking, permutation importance; to see a feature’s effect shape, PDP, escalating to ICE when you suspect interactions. For the deep mechanics of computing and plotting SHAP values, return to the SHAP lesson; to judge whether an explanation is even trustworthy on your task, the metrics lesson is the companion.

Quick check