🏢 State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
·2754 words·13 mins·
loading
·
loading
Natural Language Processing
Large Language Models
🏢 State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
Extend LLMs context via a simple, training-efficient positional encoding method, CREAM, outperforming existing methods by focusing on crucial mid-context information.