EMNLP 2026 Tutorial · Causal Foundations for Interpretability

A control task shows what a probe can score by memorising

In Hewitt and Liang's control task, each word type is given a fixed random tag, so the only way to score well is to memorise; the same linear probe still reaches 71.2%, against 97.2% on part of speech.

Your guess: control-task accuracy50%

Probe:: a linear classifier trained on the frozen activations to predict each word's tag: its part of speech (noun, verb, …) or, in the control task, its random tag.
ELMo layer 1:: the activations of the first layer of ELMo, a pretrained language model.
Control task:: each word type (a distinct word, so every “the” alike) gets one fixed random tag, so the only way to score well is to memorise.
Selectivity:: part-of-speech accuracy minus control-task accuracy, in percentage points; this gap, not the 97.2%, is what tells you about the representation.