Explain a prediction without inventing causality

An explanation method can describe how a fitted model used features without proving that those features causally produced the real-world outcome.

The idea

Explaining which clues influenced a decision is not the same as proving those clues caused the event in the real world.

Feature attribution, permutation importance, and local explanation methods describe model behavior under their assumptions. They should not be interpreted automatically as causal effects.

How it works

1. First verify that the predictive model itself performs adequately on held-out data.
2. Choose an explanation question: global reliance, local contribution, dependence, counterfactual, etc.
3. Apply a method whose assumptions match that question.
4. Check stability, correlated features, data shift, and whether the explanation supports only predictive or genuinely causal claims.

Permutation importance idea

If shuffling feature j breaks useful information and reduces held-out score, the fitted model depended on that feature on that dataset. Correlated features can make the result harder to interpret.

importance_j = score(model, X, y) − score(model, permute_j(X), y)

Check your understanding

What does a feature attribution primarily explain?

• How the fitted model used the feature
• A guaranteed real-world causal effect
• The user's intent
• The server configuration

A common misconception

Feature importance is not causal importance. A model can rely on a proxy, a confounder, or a correlated substitute.

Open Explainable AI lab