🏢 EPFL, Lausanne, Switzerland
A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention
·2455 words·12 mins·
loading
·
loading
Large Language Models
🏢 EPFL, Lausanne, Switzerland
A solvable model reveals a phase transition in dot-product attention, showing how semantic attention emerges from positional attention with increased data, explaining the qualitative improvements in l…