Same Loss, Different Story

Schematic. Grey: reconstruction. Orange: sparsity, with co-occurring concepts. Purple: one more assumption.

Rotate the learned axes, θ 20°

θ: angle between the learned axes (dashed) and the true concept axes (solid); at 0° and 90° they coincide, up to swapping names.
Reconstruction: how well the learned features rebuild the activations. Sparsity: how few features each input uses. Co-occurring concepts: no input carries a single concept.
Paired inputs: two inputs that differ in only a few concepts, the one more assumption behind the purple curve.