Striking Claim Regarding the Internal Structure of Artificial Intelligence Models

AI researcher Nick Cammarata has written an approximately 26,000-word article arguing that AI models are not entirely impenetrable black boxes and that their neurons have a cleaner structure rather than being complex nodes. This study, which has caused a stir in Silicon Valley, received praise from figures in the industry, including OpenAI CEO Sam Altman, while being met with reservation by some experts.

Former OpenAI employee and AI researcher Nick Cammarata, involved in the startup Goodfire, has published a new article on how artificial intelligence models think. This text, consisting of approximately 26,000 words, was evaluated by OpenAI CEO Sam Altman and other prominent researchers as a significant moment for understanding alignment and why AI agents and chatbots might bypass safety shields.

Contrary to popular belief, Cammarata suggested that AI neurons do not exhibit superposition and are not fundamentally uninterpretable; rather, the neurons are actually clean. For example, he stated that a neuron that activates for both a cat's face and a car is not a combined concept neuron, but instead responds to a common V-shaped pattern.

On the other hand, Google DeepMind researcher Neel Nanda stated that the study is interesting but does not provide a major update to the overall picture of interpretability, and that superposition remains valid. It was also reported that the findings have not yet been tested or peer-reviewed.