Mechanistic Interpretability

Mechanistic interpretability (sometimes abbreviated as mech interp, mechinterp, or MI) is a subfield of research within explainable AI that aims to understand the internal workings of neural networks by analyzing their concrete structures, algorithms and circuits. This approach seeks to analyze neural networks in a manner similar to the reverse engineering of conventional software.[1][2] https://en.wikipedia.org/wiki/Mechanistic_interpretability

• The term "mechanistic interpretability" was coined by Chris Olah, co-founder of Anthropic, as a description of his work in circuit analysis as opposed to usual methods in interpretable AI. Circuit analysis attempted to completely characterize individual features and circuits within models, while the broader field tended towards gradient-based approaches like saliency maps.[3][2] Before circuit analysis, work in the subfield combined various techniques such as feature visualization, dimensionality reduction, and attribution with human-computer interaction methods to analyze models like the vision model Inception v1.[4][5]
• A circuit in a neural network is composed of causal chains of feature activations. By mapping out what circuits lead to what downstream consequences, as well as by activating and inhibiting circuits, one can analyze how a neural network (such as an LLM) reaches a given result from a given input