Interpretability

LLMs’ reasoning is decoded via sparse autoencoders, revealing key features that, when steered, enhance performance. First mechanistic account of reasoning in LLMs!

MoE-X: An intrinsically interpretable Mixture-of-Experts language model that uses sparse, wide networks to enhance transparency.

LLMs use punctuation in context memory, surprisingly boosting performance by using seemingly trivial tokens.