Mechanistic interpretability breakthrough: Goodfire says neural networks are far more legible than previously thought

Neural networks are far more legible than researchers thought

Nick Camerata, who led OpenAI's mechanistic interpretability work before founding the research lab Goodfire, says the field's pessimism about understanding neural network internals was premature. His early work at OpenAI found only about 2% of neurons could be interpreted with the tools available then. With modern automated techniques and closer inspection, the picture has inverted entirely.

The vast majority of neurons in models up to trillions of parameters are now legible, Camerata says, and their internal circuits are "beautiful." Many neurons previously thought to be tangled and arbitrary turned out to be misunderstood. The technical roadblocks he expected mostly don't exist.

Sparse autoencoders—the previous approach to decoding neural networks by treating them as compressed versions of larger models—may no longer be necessary. Rather than needing to expand a model conceptually to find individual feature-detecting neurons, researchers can now examine neurons directly and determine what they do.

Why this matters for safety and capability

Interpretability cuts both ways. Better understanding of how models work helps with alignment and safety work—Camerata frames this as the foundational problem, arguing that meaningful alignment work requires knowing what you're building. But legibility also enables capability improvements. If you can identify exactly how a model performs a particular task, you can optimize and improve that specific function.

Camerata believes ambitious interpretability will be solved before it becomes critical, possibly soon, as automated science tools continue improving. Roon Kanda, commenting on the work, called it potentially "a major moment in scientific history"—significant enough that even Eliezer Yudkowsky may have to adjust his probability estimates for doom.