Reading a model's mind: sparse autoencoders explained

At 3:07 a.m., a red-team prompt makes your language model produce a strangely specific answer. It mentions the Golden Gate Bridge even though the user asked about California tax forms. The output tells you there is a bug. It does not tell you whether the model stored a “Golden Gate Bridge” concept, followed a word association, copied a training pattern, or wandered into a plausible sentence by accident.

That distinction matters when the model is doing something more serious than naming landmarks. A refusal may come from a robust safety representation, a shallow phrase match, or a fragile instruction buried in the prompt. A claim that the model used a tool may reflect an actual tool trace, or merely the model’s ability to describe what tool use would look like.

For most of the history of deep learning, we investigated these questions from the outside. We measured outputs, wrote probes, and stared at loss curves. Useful, but incomplete. Sparse autoencoders, or SAEs, are the most promising attempt yet to inspect the internal concepts that produce those outputs.

My view is deliberately less glamorous than the usual headline: SAEs are not a model mind-reader. They are a microscope and a dictionary. Used carefully, they turn an unreadable activation into testable hypotheses about what a model represents and how it behaves. Used carelessly, they turn a few attractive examples into a story with scientific-looking typography.

Why neurons do not give you the answer

A transformer stores information in activation vectors: long lists of numbers produced at each layer and token position. A d_model-dimensional residual-stream activation—the running vector passed between transformer blocks—has d_model coordinates, or channels.

For example, when d_model = 4,096, that vector has 4,096 channels. An MLP layer has its own neuron or unit activations. Neither is guaranteed to correspond to a concept.

The tempting picture is simple. One unit or coordinate means “dog.” Another means “Python.” Another means “anger.” That picture is mostly wrong.

A unit or coordinate is a scalar value, not a concept label. An MLP unit can respond to several unrelated patterns. Conversely, one concept can be represented by a direction spread across many units or channels. Researchers call the first problem polysemanticity, meaning that one unit responds to multiple meanings.

One proposed reason for this mess is superposition. It is a regime in which a model may encode more sparse features—recurring patterns the model can represent—than it has available dimensions by overlapping their directions.

Imagine a four-dimensional space trying to represent eight recurring features: Golden Gate Bridge, French text, code syntax, prime numbers, anger, gratitude, sailing, and tax forms. Giving every feature its own independent axis would require eight dimensions. Instead, the model may store sparse combinations of directions.

Here is a toy version. Suppose the Golden Gate Bridge direction is g = (1.0, 0.8, 0, 0) and the code direction is c = (0, 0.8, 1.0, 0). When both are present, the activation might look like x = 2g + 1.5c, or approximately (2.0, 2.8, 1.5, 0). The second coordinate rises for both concepts. Looking only at that coordinate, you cannot tell which idea caused it.

Overlap can be an efficient coding strategy, but it also introduces interference: one direction can affect another, making features harder to recover and interventions less clean. We are the ones left holding the tangled headphones.

A hypothetical raw-neuron example makes the distinction concrete: a researcher might find an MLP unit that activates for text about the Golden Gate Bridge, but the same unit may also activate for unrelated geographical names or a particular sentence structure. That does not mean the bridge concept is absent. It means the concept is not obliged to live in one unit.

The widely cited Golden Gate Bridge result is a separate example. Anthropic identified an SAE feature in Claude 3 Sonnet associated with the bridge; it was not a raw neuron. That feature is useful evidence about a learned direction, not proof that one original model unit means “Golden Gate Bridge.”

To understand the model, we need a better coordinate system.

What a sparse autoencoder actually does

An autoencoder is a model trained to compress an input and then reconstruct it. A sparse autoencoder adds a training pressure: the internal code should use only a small number of active features for each input.

Take an activation vector x with d coordinates. The SAE maps it to a feature vector f with m coordinates, where m is deliberately much larger than d. This is an overcomplete dictionary: there are many possible feature directions, but only a few should be selected for any one activation.

The value of m is an SAE design choice, not a census of how many ground-truth features the language model contains.

The easiest mental model uses three schematic equations:

f = ReLU(W_e(x - b_x) + b_e)

x_hat = W_d f + b_d

loss = ||x - x_hat||² + lambda ||f||₁

These equations are schematic, not a complete specification of a production SAE. W_e and W_d are the encoder and decoder weight matrices; b_x, b_e, and b_d are biases. W_d[:,j] denotes the decoder column for feature j.

The encoder produces f, the sparse feature activations. ReLU sets negative values to zero in the common version of the architecture. The decoder turns those features back into a reconstructed activation, x_hat.

The first term rewards accurate reconstruction. The final term is an L1 penalty, meaning the sum of the absolute values in f; it penalizes total activation magnitude. It encourages sparsity, but it does not directly count how many features are nonzero or impose a fixed active-feature limit. lambda controls how painful that penalty is.

If an implementation uses TopK sparsity, it typically keeps a fixed number of the largest feature activations instead. An L0-style constraint or penalty targets the number of nonzero features directly. Those mechanisms are not interchangeable with L1, even though all of them can produce sparse-looking codes.

The schematic objective also has a scaling degeneracy if decoder columns are left unconstrained. Multiply a decoder column by a and divide its corresponding feature activation by a; the reconstruction stays the same.

For a > 1, the L1 contribution shrinks. Practical implementations prevent this with decoder-column normalization, for example ||W_d[:,j]||₂ = 1, or with another regularizer. Check which choice an implementation makes before comparing SAEs.

Return to the toy four-dimensional activation. An SAE might have eight features and produce:

f = (0, 2.0, 0, 0, 0, 0, 1.5, 0)

Only feature 2 and feature 7 are active. Their decoder directions combine to reconstruct the original vector.

We might eventually discover that feature 2 activates for Golden Gate Bridge contexts and feature 7 for code contexts. The original activation coordinates did not offer that clean separation. The expanded sparse code does.

This works because sparse explanations are more constrained than dense ones. Without a sparsity penalty, the SAE can spread information across many features and learn an arbitrary rotation of the activation space. Many different dictionaries could reconstruct the same data.

Sparsity asks the system to prefer explanations that reuse a small number of features at a time.

That pressure is useful, not magical. If the penalty is too weak, the representation remains dense and difficult to interpret. If it is too strong, the SAE throws away meaningful detail and keeps only broad, high-frequency patterns.

Better reconstruction and cleaner features are competing goals. There is no single setting that gives you perfect fidelity and perfect human concepts.

The decoder columns are also worth understanding. A feature is not a new neuron inside the original language model. It is a direction in the original activation space, learned by the SAE.

When feature f_j is active, its decoder vector contributes to the reconstructed activation. That is why a feature can correspond to a concept spread across hundreds of original channels.

The early dictionary-learning work showed that this approach can produce features that humans can interpret more easily than individual neurons. That result is the foundation of modern SAE research (the original monosemanticity work).

“Monosemantic” is a hypothesis, not a certificate

A feature is called monosemantic when its activations appear to represent one coherent concept rather than a grab-bag of unrelated concepts. The word is useful, but it can encourage overconfidence.

Researchers usually inspect the examples that produce the highest activation for a feature. If the top examples include paragraphs about the Golden Gate Bridge, bridge construction, and photographs of the bridge, “Golden Gate Bridge” is a reasonable provisional label.

The label is still supplied by a human. It is not printed inside the feature.

A careful investigation asks harder questions:

• Does the feature activate for the bridge in Spanish?
• Does it activate when the bridge is mentioned only indirectly?
• Does it respond to other famous suspension bridges?
• Does it fire because of the concept, or because the text contains a particular name followed by a particular punctuation pattern?

Negative examples matter as much as positive ones. If a feature activates for every paragraph containing “San Francisco,” it is not specifically a bridge feature. If it activates for the bridge but only at the token position where the name appears, it may be a lexical detector rather than a broad geographical concept.

This is where a feature dashboard can mislead. Ten attractive top examples can make a feature look beautifully coherent. The next 10,000 examples may reveal that it also responds to airport codes, fog, and the word “gate” in software documentation.

The right workflow treats a label as a hypothesis. Write down what should activate the feature and what should not. Then test both.

From a pretty label to a causal result

Seeing a bridge feature activate is interesting. Showing that it matters is much stronger.

From activation to intervention

An SAE feature is not a native variable exposed by the language model. It is a coordinate in the SAE’s code.

To intervene, take the activation x at the exact layer and token position, encode it to f, and edit the chosen feature value to form f'. Setting f'_j to zero suppresses feature j; assigning another value amplifies or reduces it.

If you replace the entire activation with the SAE reconstruction, patch x' = b_d + W_d f' into the language model at that same hook. If you want to preserve the SAE’s unexplained residual, first define r = x - (b_d + W_d f).

Then patch x' = x + W_d(f' - f). The second form changes the chosen decoder contribution while retaining the original residual.

An arbitrary increase in f'_j can still push the model off-distribution, so a dramatic output is not automatically a meaningful causal result.

If suppressing the feature makes the model less likely to mention the bridge, and increasing it makes the model more likely to discuss the bridge, that is evidence the feature participates in the behavior. It is no longer just a label attached to a visualisation.

The evidence is not automatically decisive. An intervention can damage several correlated computations at once. Adding a decoder direction may push the activation into a region the original model never visits. A downstream layer may ignore the feature in one context and amplify it in another.

Testing causal claims

A stronger test uses controls:

• Compare the targeted feature with a randomly selected feature of similar frequency.
• Test both on prompts that mention the bridge and prompts that do not.
• Measure the intended effect, such as bridge-related completions, alongside side effects, such as answer quality, refusal rate, or unrelated geographical answers.

Anthropic’s work applied SAEs to Claude 3 Sonnet and reported millions of features with interpretable patterns, including features associated with concrete entities, programming concepts, and behavioral patterns.

Crucially, the work also used feature interventions that changed model behavior in the predicted direction. That moved SAEs beyond “here is a neuron that looks interesting” toward “here is an internal variable we can manipulate and test.” The work is described in Scaling Monosemanticity.

The next step is to follow features across layers. A feature that appears at one layer tells you about a state. It does not by itself tell you what computation produced that state or how the model used it.

Attribution graphs and circuit-tracing methods try to connect features across the forward pass: one feature detects a bridge, another retrieves a fact, another shapes the answer.

That is closer to explaining a mechanism. It is also much harder.

The caveats that survive the demo

Four caveats

1. Reconstruction loss. An SAE does not preserve every detail of the original activation. In the Claude 3 Sonnet paper’s layer-12 residual-stream experiment, replacing the layer-12 activations with SAE reconstructions increased the evaluated next-token language-model loss by approximately 10–40%, depending on the experiment.

   This is a percentage increase in model loss, not a 10–40% drop in task accuracy or task performance. Loss and accuracy measure different things, so one cannot be converted into the other by reusing the percentage.

   It is also not the SAE’s reconstruction mean squared error, or MSE—the average squared difference between x and x_hat. MSE measures how closely the SAE reproduces the activation; model loss measures what happens after the downstream network receives that activation.

2. Feature splitting. One broad concept may be represented by several SAE features: one for the bridge in factual prose, another for the bridge in image descriptions, another for construction details. A wider SAE may split a feature that a narrower SAE keeps together.

   This can improve interpretability, or simply give you more opportunities to invent labels.

3. Absorption. In SAE work, absorption refers to a feature-recovery failure: a real model feature, especially a rare or weak one, may be represented through existing SAE features or a combination of them, so the SAE does not recover it as its own feature.

   That is different from one learned feature bundling several patterns. The latter is polysemanticity, also called feature impurity. “Deception” may actually mean a narrow textual style associated with deceptive examples. It may miss deception expressed through a different strategy. A feature name is never broader than the tests supporting it.

4. Distribution. An SAE is trained on activations from particular prompts, token positions, layers, and model versions. A feature discovered at one layer of one checkpoint is not automatically meaningful at another layer or after a fine-tune. A production model can also learn to represent the same behavior differently after an update.

The most visible failure mode has a simple symptom: the top examples look perfect, but interventions do nothing on held-out prompts. That usually means the feature label captured a correlation in the inspection set, the activation was not causally important, or the intervention was applied at the wrong position.

The opposite failure also occurs: changing a feature produces dramatic behavior, but the model’s general quality collapses because the patch was an unnatural perturbation.

Finally, SAEs do not eliminate the need to understand the surrounding architecture. The activation site matters. A feature at an early layer may represent a raw lexical pattern; a feature later in the network may reflect a conclusion assembled from many earlier computations.

To interpret an SAE result, you need to know where it was measured in the transformer’s forward computation.

The strongest objection, and the right answer

The strongest objection is practical: why spend weeks decomposing activations when black-box evaluations, red-teaming, and output monitoring already tell you whether the model behaves badly?

That objection is fair. SAEs are expensive to train and inspect. Their labels are partly subjective. Their reconstructions are incomplete. A good behavioral evaluation can catch a failure without explaining its cause.

The answer is not that internal analysis replaces behavioral testing. It does not. The answer is that behavior alone often underdetermines the mechanism.

Two models can produce the same safe answer for different reasons. One may understand the instruction and apply a robust constraint. Another may rely on a phrase match that fails after a small paraphrase.

If an SAE identifies different internal features in those cases, it can suggest the next test. It can also help investigate failures before they become obvious in outputs, although that promise remains an empirical claim rather than a guarantee.

Use LLM evaluations to measure behavior. Use SAEs to generate and test hypotheses about why the behavior occurs. Neither is sufficient alone.

What to do on Monday morning

Design the test

Start with one narrow question that could be falsified. “Does the model represent the Golden Gate Bridge as a reusable concept, or does it merely react to the phrase?” is better than “Can we understand the model?”

Freeze the model checkpoint, tokenizer, activation layer, and token position. Collect a few thousand prompts covering positive examples, paraphrases, related concepts, and deliberately unrelated uses.

If you use a released SAE, make sure it was trained for that model and activation site. If you train one, record the dataset and sparsity settings as carefully as you record the model version.

Inspect the highest-activating examples, but do not stop there. Write a provisional label and a small test suite. Include bridge mentions, other suspension bridges, the word “gate” in software, translations, negations, and prompts where the bridge is implied but not named.

Intervene and record

Then intervene. Compare the baseline with suppression and amplification of the candidate feature. Hold the prompt set fixed. Include a random-feature control.

Measure the behavior you care about and the side effects you would regret shipping. A handful of striking completions is not an evaluation.

Keep the SAE checkpoint, hook location, feature index, activation normalization, reconstruction error, and held-out results together. When the model changes, rerun the tests. The most dangerous sentence in an interpretability notebook is “works on my favorite prompt.”

For a production system, integrate the results with ordinary model observability, traces, and output evaluations. Do not make a single SAE feature your only safety gate.

A model can evade a monitor by changing its internal representation, and your SAE can miss the relevant feature entirely.

The promise of sparse autoencoders is still enormous. They give us a vocabulary for internal states that were previously visible only as columns of numbers. But a vocabulary is not a theory, and a theory is not a proof.

The useful question is no longer “Can we read the model’s mind?” It is more disciplined: “What internal feature might explain this behavior, and what intervention would prove us wrong?” SAEs are the best tool we have for asking that question at scale. They become trustworthy only when we keep trying to break the answer.

Check your understanding