Every interpretability claim sits on Pearl's causal ladder
Claims that a circuit computes a function, that a head mediates a behaviour, or that an SAE feature controls a capability are claims about causal influence inside the model. The paper's formal position follows: interpretability claims should be stated in the language of causal inference and identifiability — specify the estimand, the intervention class, and the equivalence class implied by the available evidence. Each rung of the ladder answers a different kind of question and requires strictly stronger evidence — evidence at rung k does not license claims at rung k+1. Expand each rung to see what its evidence licenses for the running example, refusal.
L1 · AssociationDo activations correlate with refusal?
Diagnostic: is the system manipulated, or only observed? If the method only computes statistics from forward passes, the evidence is associational — even if the summary is worded causally. A correlation between activations and refusal is equally consistent with a confounder driving both.
Typical methods: linear probes, activation logging, SAE feature attribution, logit lens, PCA. Estimands: p(y|x), p(h|x), p(y|h).
L2 · InterventionalCan we induce refusal by manipulating activations?
Diagnostic: is the manipulation externally controlled? Clamping, patching, ablating, or steering the forward pass — with the modification set by the experimenter — yields interventional evidence: p(y|do(·)). A caveat the paper stresses: interventions do not act in isolation; the observed effect can reflect the patch source, injection method, or strength rather than the variable itself.
Typical methods: activation patching, causal tracing, ablations, steering vectors, ROME/MEMIT edits.
L3 · CounterfactualFor this jailbroken prompt, what change would have caused refusal?
Diagnostic: is the query anchored to a specific observation? Counterfactuals (yv←v') reason about the same instance under an alternative intervention, which requires Pearl's abduction–action–prediction recipe. Standard activation patching performs only the action step — which is why most “counterfactual” claims in interpretability are actually L2 claims narrated counterfactually.
Rarely reached: counterfactual patching, causal scrubbing, necessity tests, uniqueness proofs.
Answers consistent with the evidence
“The model encodes refusal in this layer.”
Take this everyday sentence at face value and it makes two commitments the usual evidence does not close. In the paper these two factors are called the rung mismatch and the identification gap; the pilot audit below found them structuring roughly half of the claims in the literature, often compounding.
Does decodability imply use?
Why this gap matters, and what closes it
Probe success establishes that some direction predicts the concept — decodability, not model use. The information may be present but unused by the model's own computation: control tasks show probes can fit random labels, and amnesic probing found that removing part-of-speech information from BERT barely affected its language modelling, despite >97% probe accuracy.
What closes it: interventional evidence. Intervene on the candidate direction and measure the behavioural change — moving from “can we decode it?” to “does the model use it?”.
Is the direction unique?
Why this gap matters, and what closes it
Many probes and sparse factorisations achieve near-identical objective values through different directions — the evidence identifies an equivalence class, not a single axis. Identifiability theory proves that without additional structure, unsupervised recovery of unique latents is impossible; sparsity is a useful inductive bias, not an identification guarantee.
What closes it: identifying structure — sufficient variability across contexts (SSAEs use paired samples of concept shifts), interventional data, or constraints like independence that pin down the axes up to permutation and scaling.
Six recurring versions of the same mistake
Steering ≠ Encoding
αv and shifting behaviour is L2 evidence of a control handle. Claiming the model “represents honesty” asserts v is an internal causal variable — but v may entangle concepts, be non-unique, or exploit distributional shortcuts. Resolve with transfer across distributions, subspace specificity, and compositional tests.Decodability ≠ Model Use
Local Sensitivity ≠ Global Importance
Existence ≠ Uniqueness
Alignment ≠ Ontological Identity
Subspace ≠ Coordinates
Rotate the learned basis — the reconstruction loss won't notice
A sparse two-dimensional source, mixed and then learned back by an autoencoder. Drag θ, the rotation of the learned basis away from the true axes. Reconstruction is blind to θ: for any invertible remixing of the latents, the composed encoder–decoder achieves exactly the same objective. Adding only a sparsity penalty prefers some angle — the wrong one. An identifiable model has its minima exactly at the true axes, 0° and 90°.
This playground reproduces the θ figure designed for the printed poster — a stylised illustration, not an experiment reported in the paper. What it illustrates is the paper's identifiability argument: any invertible transformation of the latents leaves the reconstruction objective unchanged, so per-latent semantics (“latent i is refusal”) are not stable without identifying structure — and proxies like reconstruction error or sparsity cannot select among the equivalent solutions.
Two iron filings, one difference you cannot see
One of these filings is magnetised. Look at them, weigh them, lift them — identical. Only a new interaction separates them. In the paper's framework, concepts are reframed as affordances: what an interpreter can discover depends on the interactions available to them, not on presumed ground-truth labels.
What a method can identify is bounded by the interactions it affords.
Where identifying structure actually comes from
Three recurring sources in the representation-learning literature, each a family of “magnets”:
Interventional data. Paired samples before and after perturbations, temporally resolved actions, or multi-environment data force competing explanations apart, identifying latents up to equivalence classes determined by the intervention family.
Natural variation. Nonstationarity enables nonlinear ICA through segment discrimination; self- and weak supervision exploit contrasts across timesteps, augmentations, or contexts — leverage without explicit intervention.
Inductive biases. Independence assumptions, sparsity constraints, and functional restrictions on the mixing process rule out families of solutions.
In practice results combine sources, and the resulting equivalence class reflects the joint constraints of all available affordances. There is no free lunch: supervision is relocated, not eliminated — it enters through the affordances a method leverages and through downstream interpretation.
Roughly half admit stronger readings than the evidence licenses
Gap between claim rung and method rung (n = 185)
| claims L1 | claims L2 | claims L3 | |
|---|---|---|---|
| L1 evidence | 15 | 19 | 23 |
| L2 evidence | 0 | 71 | 57 |
| L3 evidence | 0 | 0 | 0 |
The six recurring elevation patterns
How the audit was run (and how seriously to take it)
Protocol. Each claim received a method rung (what the reported procedure establishes: 1 observational, 2 interventional, 3 counterfactual) and a claim rung, read off linguistic markers — “correlates with / predicts” is L1; “mediates / is sufficient for” is L2; “encodes / represents / computes / THE circuit” is L3. Gap score = max(0, claim rung − method rung). Hedged claims (“may encode”) are coded at the underlying rung with reduced confidence.
Annotators. Primary annotation by Claude Opus 4.5 with human oversight, guided by a structured codebook and five calibration anchor papers; all 186 classifications independently replicated by seven LLMs across four model families (plus DeepSeek R1 on 166 claims). Krippendorff's α: method rung 0.66 (meets the pre-specified 0.6 threshold), claim rung 0.56 (moderate), annotator confidence 0.11 (near chance). Every model independently lands in a 34–66% band. The paper is explicit: because all annotators share pretraining corpora containing the MI literature, “LLM–LLM agreement is a consistency check, not a validity guarantee.”
Human verification. 43 claims across 12 papers fact-checked against arXiv sources; 84% required no corrections.
Robustness. Gap rates are essentially uniform across abstracts (53.3%) and body text (52.9%) — a field-wide linguistic convention, not abstract-specific compression. Reclassifying the claims whose gap rests solely on definite-article or conventional-verb patterns (“THE circuit” as a name) lowers the rate from 53.5% to 47.0%; hence the robust statement is 47–54%. By paper type, mean gap rates range from circuit discovery at 0.64 (16 papers) down to applied/production work at 0.35 (7 papers).
Cost and openness. ~1,468 API calls, ~44M input tokens, ≈$139 total. De-identified annotations from all eight annotators, the codebook, calibration rationales, and pipeline code: github.com/rpatrik96/mech-interp-claim-calibration. The annotated-paper list is withheld while authors can review their own annotations; two positively calibrated papers are named, the rest appear as aggregate statistics.
The interpretability claims checklist
Seven questions that align claim language with evidential strength — from the paper's appendix, useful for writing and for reviewing. Items 2–4 work here: pick your rungs and the rung check answers itself.
Pick both rungs above.
“A paper that states its rung, names its equivalence class, and bounds its scope contributes more than one whose language implies stronger evidence than reported — even if the former's claims are narrower.” The checklist is constructive: it is not a scorecard for past work, and it is not meant to discourage exploration or raise publication barriers.
Four places where the two fields need each other
CRL provides formal tools for equivalence classes, causal levels, and transportability; interpretability supplies empirical phenomena and safety-relevant evaluation criteria.
Counterfactual semantics for safety
Safety verification asks counterfactual questions — “would this output have been harmful had we not intervened?” — while patching delivers L2 evidence.
Open: when does activation patching coincide with counterfactual conditioning in transformers? Can we build tasks where L2 and L3 answers provably diverge?
Task-relative equivalence classes
Different tasks need different identification strengths: binary classification needs only affine identification, steering needs a concept subspace (blockwise), knowledge editing needs individual elements.
Open: what is the weakest equivalence class empirically sufficient per task — and can objectives target it directly?
Compositional control
Model edits do not compose, while task vectors compose linearly. Establishing whether and when steering vectors compose is essential for claiming they control variables of interest.
Open: under what conditions do interventions compose additively — and can compositional failure be predicted from representation geometry?
Transportability as a theory of edit generalisation
Failures of model editing are predictable consequences of unexamined transportability assumptions; systematic edit failures are natural experiments revealing the violated ones.
Open: criteria that predict, before editing, which prompts or distributions an edit should generalise to.
Alternative views, engaged head-on
“Interpretability can be wholly pragmatic.” Iterate against measurable proxy tasks and let behavioural outcomes define progress — a direct response to unsupervised metrics telling us little.
Our reading: the criterion assumes what it seeks to avoid.
Proxy success is informative only insofar as the proxy identifies structure relevant beyond the proxy distribution — an identifiability requirement. SAE probes failing under distribution shift is a failure of identification, not of SAEs; identifiability makes it predictable rather than surprising. A reason to improve how SAEs are evaluated and constrained, not to discard them.
“Symmetries suffice to formalise interpretability.” Explanations should be invariant to transformations that preserve model behaviour.
Our reading: complementary, with two tensions.
Symmetry constraints prescribe which transformations explanations should be invariant to; identifiability characterises which representations the evidence can actually distinguish. But the path from symmetry violation to concrete empirical breakdown is underspecified, and privileging human-interpretable symmetries risks a streetlight effect: model-relevant structure that violates them may be systematically overlooked.
Frequently asked questions
What's actually new here? Hasn't the field already said much of this?
Largely yes, and the paper says so: “our contribution is not to discover these issues but to provide a unified causal framework that makes them precise.” The community has flagged necessity/sufficiency testing, entanglement, and evaluation rigour; causal abstraction, DAS, and causal scrubbing already reason causally. What the framework adds is the shared bookkeeping — estimand, intervention class, equivalence class, transportability — so that the same words stop referring to different quantities across papers, plus the diagnostic machinery (ladder placement, identifiability conditions, the checklist) and the pilot audit quantifying the claim–evidence gap.
Is counterfactual (L3) evidence even achievable for real LLMs — or is this an impossible bar?
It is genuinely hard: counterfactual claims remain largely unverifiable without controlled supervision, and the audit found no L3-grade evidence among 186 claims. The paper treats that as a diagnosis, not a demand that every paper reach L3 — different rungs answer different questions, and most goals are well served by honestly-labelled L1/L2 evidence. Reaching L3 is the first Call-to-Action direction: determine when activation patching coincides with counterfactual conditioning in transformers, find the minimal exogenous annotations that restore counterfactual semantics, and build tasks where L2 and L3 answers provably diverge.
We have full white-box access to weights and activations. Don't we already have every affordance we need?
Access is not the same as identifying interaction. Reading every weight gives you unlimited L1 evidence, but equivalence classes are broken by variation — interventions, environments, contexts — not by resolution. And interventions do not act in isolation: a patching effect can reflect the patch source, injection method, or strength rather than the variable itself. The magnet in the demo is not “more access to the filing”; it is a different interaction that makes a previously invisible property visible. The open question for any method is which variations it actually induces, because those determine what it can identify.
So is the single-refusal-direction result (Arditi et al., 2024) wrong?
No — that is exactly the distinction the framework is for. Inducing and bypassing refusal by manipulating a direction is strong L2 evidence of a sufficient control handle; the framework only cautions against the further readings that this direction is unique (identifiability) or that the model uses it in unmodified forward passes on new distributions (transportability). Those require ruling out equivalent directions and testing transfer — which is what the “one claim, two gaps” panel above unpacks on this very example.
What exactly do you mean by “identifiability”?
An estimand is identifiable up to an equivalence class if it is constant within the class but varies across classes — so estimating it determines the class, not the single hypothesis inside it. It is a statement about what structure is invariantly recoverable given your evidence, not about what entities exist. Without it, any invertible remixing of latent coordinates achieves the same objective value, so “latent i is refusal” is not stable: an equally good solution blends refusal with something else across components.
What is an estimand, and why call them “implicit”?
The estimand is the quantity a method actually estimates, as opposed to the question it is recruited to answer. Interpretability methods rarely state them: saliency maps are recruited to answer “which features caused this output?” but estimate local input–output sensitivity relative to an analyst-chosen baseline; probes are recruited to answer “does the model represent this concept?” but estimate the existence of a decision function in a restricted probe class. The gap between the recruited question and the actual estimand is where claims inflate.
My probe reaches 95% accuracy. Doesn't that show the model represents the concept?
It shows the concept is decodable from that layer — L1 evidence. Decodability ≠ encoding: the information may be present but unused. Control tasks show probes fit random labels, and amnesic probing found that removing POS information from BERT barely affected language modelling despite >97% probe accuracy. To move from “can we decode it?” to “does the model use it?”, intervene on the direction and measure the behavioural change.
Where does activation patching sit on the ladder?
L2 — genuinely interventional. But it identifies the existence of a sufficient path, not uniqueness or necessity: multiple circuits can produce identical patching effects. Necessity is an L3 property requiring a structural model; standard patching performs only the “action” step of Pearl's abduction–action–prediction recipe. Testing both denoising (sufficiency) and noising (necessity) is the practical upgrade the framework grounds.
Do SAEs find the model's true features?
An SAE's estimand is a factorisation minimising reconstruction error under sparsity — and any factorisation satisfying the constraint lies in the same equivalence class. The “true features” are not identified; you recover what the objective and its inductive biases reward. Sparse factors need not be disentangled, causally operative, or concept-aligned, and auto-interp scores can be high even on random structure. The fix is identifying structure (e.g., sufficient variability across contexts, as in SSAEs), not abandonment.
Isn't pragmatic interpretability enough? The method works on the proxy task.
Proxy-task success is informative only insofar as the proxy identifies structure relevant beyond the proxy distribution — which is an identifiability requirement in disguise. A proxy justifies stronger conclusions only when it rules out alternative internal explanations that would also solve the proxy yet fail on the target. Identifiability makes distribution-shift failures predictable rather than surprising, and names the assumption to check.
But steering vectors do work. Why demand guarantees?
The evidence is remarkable — and not guaranteed to generalise: model edits revert on paraphrases, steering vectors miscalibrate out of distribution, probes fail under shift. Fine for exploration; unavoidable to fix for safety-critical deployment, where you must ask why something works and when it will stop working. Identifiability makes that answerable: it specifies which assumption to verify and what failure to expect if it breaks — ruling out failure modes before deployment rather than after.
Doesn't CRL assume ground-truth latent variables that LLMs simply don't have?
Classical CRL does; the paper's move is to reframe concepts as affordances. Instead of asking whether a representation recovers true latents, ask what it affords for specific interactions — ground-truth variables become formal reference variables specifying which distinctions are recoverable from a given interface. The magnet demo above is the intuition: what can be identified is bounded by the variations the system affords. Models may even encode structure humans lack vocabulary for.
How did the audit decide a claim “admits a stronger reading”?
Each claim got a method rung (what the procedure establishes) and a claim rung read off linguistic markers — “correlates with” is L1, “mediates” is L2, “encodes / represents / THE circuit” is L3. Gap = max(0, claim − method). Hedged claims are coded at the underlying rung with reduced confidence, because the study deliberately measures surface-level interpretive risk — what a skimming reader could take away — not author intent.
So is half of mechanistic interpretability wrong?
No, and the paper says so directly: the gap rate is evidence that the field's linguistic conventions have not yet converged on vocabulary that tracks causal evidence strength — not evidence that findings are untrustworthy. One heavily rung-elevated calibration paper has been independently replicated by multiple groups: rung-elevated language can coexist with strong empirical support. And “THE circuit” is often a naming convention inherited from neuroscience — the sensitivity analysis that treats it as such is exactly why the robust range is 47–54%.
Which methods and paper types show the largest gaps?
By paper type (mean gap rate): circuit discovery 0.64 (16 papers), knowledge localisation 0.50 (3), other 0.47 (19), evaluation/benchmark 0.45 (5), applied/production 0.35 (7). The dominant single pattern is L2 patching evidence narrated in L3 mechanistic language. Engineering-framed applied papers score best — system-level success framing avoids the mechanistic vocabulary that is the primary source of rung elevation.
What kind of data or structure buys identifiability?
Three recurring sources: interventional data (paired pre/post-perturbation samples, temporal actions, multi-environment data), natural variation (nonstationarity enabling nonlinear ICA; self- and weak supervision exploiting contrasts across timesteps, augmentations, contexts), and inductive biases (independence, sparsity, functional restrictions on the mixing). Each breaks equivalence classes; real results combine them, and the class you end up with reflects the joint constraints of all available affordances.
How much identifiability does my task actually need?
It is task-relative. Binary classification needs only affine identification (linear separability survives); steering needs blockwise identification (the concept subspace); knowledge editing needs elementwise identification. Element-wise is strongest and subsumes the rest, but demanding it everywhere is overkill — an open problem in the paper is finding the weakest class empirically sufficient per task and designing objectives that target it directly.
What should I do differently in my next paper?
Follow the recipe: state the estimand; place the question on the ladder; check which evidence your method actually provides; then match the language — “decodable” for probing, “a sufficient control handle” for patching, not “represents” or “THE mechanism”. Test denoising and noising for patching claims, verify variability assumptions for SAE-style methods rather than reporting a low objective, and run the checklist above before submitting.
Paper, blog, animations, checklist
Scan straight off this screen — the same four codes as on the printed poster.
Cite
@misc{joshi2026causality,
title = {Position: Causality is Key for Interpretability Claims to Generalise},
author = {Joshi, Shruti and Mueller, Aaron and Klindt, David and
Brendel, Wieland and Reizinger, Patrik and Sridhar, Dhanya},
year = {2026},
eprint = {2602.16698},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2602.16698},
}