interpretability

Without changing a single model parameter, this technique directly adds a direction vector representing a specific concept or behavioral tendency into a model's internal activation signals at the moment of inference, changing the model's behavior in real time. This sounds like a natural next step for interpretability research — now that you've found a feature, why not just steer it directly — but a growing body of research shows this "scalpel" cuts into places it shouldn't more easily than expected.

An AI model can produce a correct answer, yet no one — not even the engineers who built it — can explain, in human-readable terms, why it arrived at that answer. This gap between "capability" and "explainability" is the <a href="/en/glossary/interpretability/black-box-problem/">Black Box Problem</a>, and it's currently the most fundamental obstacle to auditing, debugging, and holding AI systems accountable.

A small trained neural network that decomposes an AI model's tangled, hard-to-interpret internal neuron activations into a large set of "features," each corresponding to a single, identifiable concept — currently one of the primary tools in mechanistic interpretability research.