Episode Details

Most of what we currently call "feature discovery" in language models is wrapped up in dictionary-learning methods like sparse autoencoders (SAEs) – which work, and which have been scaled to millions of features on frontier-scale models, but which bundle two distinct commitments into a single training objective: a reconstruction loss and a sparsity loss over a fixed size dictionary. Those commitments make sense if your goal is reconstructive decomposition – if you want to take an activation and rebuild it from a sparse code. They make less obvious sense if your aim is to find interpretable structure (directions? features?) in activation space, to retrieve representative examples, identify causal interventions, or measure how representations change across layers and inputs. And it turns out a lot of that doesn't really need the full SAE machinery.

An Exemplar Partitioning dictionary built from Gemma-2-2B L12 activations at p2 (K = 5,129). Left: eight sample regions, each shown with its member count, its exemplar's logit-lens [nostalgebraist, 2020] decode, and an excerpt of a member input with the activating tokens highlighted. Right: a PCA-projected 3D rendering of the Voronoi partition; each cell is one region, with a random selection also labelled with logit-lens decode.

This [...]

---

Outline:

(02:08) Glossary

(03:45) Exemplar Partitioning

(06:09) Inference

(06:51) Properties of the EP dictionary

(07:29) Concept detection (AxBench)

(08:16) How EP and SAEs relate

(09:40) Find and steer refusal

(11:37) A free OOD signal

(12:45) Cross-checkpoint drift (base ↔ IT)

(14:58) Domain saturation

(15:59) Inside the partition

(17:33) Future work

(21:24) Thats all for now

---

First published:
May 16th, 2026

Source:
https://www.lesswrong.com/posts/RroeHBSkBXXDsrryq/an-introduction-to-exemplar-partitioning-for-mechanistic-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

---

---

Listen Now