LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization

↗ arXiv ↗ Hugging Face

TL;DR

#

Key Takeaways

#

Why does it matter?

#

LongPO offers an efficient way to enhance LLMs for long contexts by transferring short-context capabilities, reducing reliance on expensive manual annotation. This approach opens new research directions for adapting LLMs to diverse context lengths and improving alignment strategies.

Visual Insights

#

🔼 This figure compares the performance of several large language models (LLMs) on both long-context and short-context tasks. The long-context performance is measured using the InfiniteBench benchmark, while the short-context performance is measured using the MMLU benchmark. The models compared include GPT-4-128K (a high-performing model), and several smaller LLMs. The figure visually demonstrates the performance gap between short and long-context tasks, showing that even high-performing, short-context models can underperform on long-context benchmarks. It highlights a key challenge addressed in the paper: the difficulty of aligning LLMs to perform well in both short and long-context scenarios.

read the caption

Figure 1: The comparison of long-context (InfiniteBench) and short-context (MMLU) performance among GPT-4-128K and smaller LLMs.

🔼 Table 1 presents a comparison of the long-context performance of the proposed LongPO method against various baselines. The models are evaluated across multiple metrics on several benchmark datasets, focusing on long-context tasks. Higher scores indicate better performance. Some results were obtained by the authors, indicated by the symbol ‘♭’, while others are sourced directly from the original benchmark papers. For a comprehensive breakdown of the RULER benchmark results, refer to Table 2.

read the caption

Table 1: Long-Context Performance of our LongPO compared with baselines. Higher is better for all metrics. Results marked with ♭♭\flat♭ are evaluated by ourselves, while other results of baselines are sourced from the original benchmarks. Full results on RULER are listed in Table 2.

In-depth insights

#

LongPO: Overview

#

While “LongPO: Overview” isn’t explicitly a heading, we can infer its core concept from the paper. LongPO likely introduces a novel alignment method targeting long-context LLMs. It aims to equip models with the ability to effectively transfer short-context capabilities to these longer scenarios. The core problem being tackled is the difficulty in aligning LLMs for long contexts due to the scarcity of annotated data and the challenges in balancing short and long-context performance. LongPO’s key innovation probably revolves around a clever way to circumvent the need for extensive manual annotation, potentially through a self-evolution or self-supervised approach, allowing the LLM to learn from its own generated data or existing short-context knowledge, and maintaining shorter context performance throughout this evolution.

Short-to-Long Align

#

The concept of ‘Short-to-Long Alignment,’ addresses a core challenge in scaling LLMs to handle extended contexts. The summary discusses transferring capabilities learned from short-context data to long-context scenarios. This focuses on maintaining performance and alignment across varying input lengths. The idea revolves around mitigating performance degradation observed in long-context tasks when models are primarily trained on short sequences. It tackles the imbalance between short- and long-context performance, aiming to retain proficiency while extending contextual understanding. Strategies involve fine-tuning and alignment techniques to ensure the model effectively utilizes information from long contexts without losing its ability to process short ones. This highlights the necessity of specialized training and optimization methods tailored for long-context data, as simply concatenating short-context data proves insufficient. A key goal is to optimize the trade-off between short and long sequence processing to enable LLMs with extended context windows.

KL Context Control

#

KL context control is a strategy to manage the model’s behavior within a specific context. This is achieved using Kullback-Leibler (KL) divergence. By minimizing the KL divergence, the model is encouraged to stay close to a prior distribution and avoid generating outputs that deviate too much. This method can prevent undesirable behaviors like hallucination or nonsensical output within particular contexts. However, there is a trade-off between control and flexibility: too much constraint may stifle the model and compromise its ability to generate creative solutions. Effective KL context control requires carefully tuning the parameters to strike a balance between adhering to the desired constraints and allowing for sufficient exploration of the solution space. The primary goal of KL context control is to improve the reliability and quality of model-generated content within a context.

512K Self-Evolving

#

The concept of ‘512K Self-Evolving’ suggests a language model’s capacity to expand its context window to 512K tokens through self-improvement. It implies a process where the model, starting with a smaller context, iteratively trains itself on data it generates, allowing it to handle longer sequences. Key aspects of such a system involve the method for creating training data, the architecture enabling longer contexts, and strategies to maintain performance on shorter contexts. Self-generation could involve tasks like summarizing long documents or answering questions that require understanding across a larger context, enabling iterative model improvement. The goal is to develop a model that can effectively leverage information within a 512K token window while avoiding performance decline on shorter context tasks.

No Ext. Annotat.

#

The paper emphasizes the importance of circumventing the need for external annotations, especially in the context of long-context LLMs. The scarcity and impracticality of human annotation for extended contexts pose significant challenges. LongPO addresses these challenges by leveraging self-generated data and internal model knowledge. The approach enables the transfer of capabilities from short-context alignment to long-context scenarios, effectively eliminating reliance on external supervision. This self-evolution property is a key contribution, offering a more efficient and scalable approach to long-context alignment. The focus on intrinsic model capabilities opens new avenues for adapting LLMs to diverse context lengths without the prohibitive costs of extensive manual annotation.

More visual insights

#

More on figures

🔼 This figure illustrates the process of creating short-to-long preference data pairs. It begins with a long document. A short chunk of text is extracted from that document. An instruction is generated based on this short chunk. Then the model generates a response based on this short chunk (chosen response) and the entire long document (rejected response). These two responses paired with the same instruction form a data pair showing preference for one over the other. This process is repeated many times to build up a dataset.

read the caption

Figure 2: The procedure of generating short-to-long preference data from step 1 to 7.

🔼 This figure compares the short-context performance of Mistral-7B-LongPO and several baseline models relative to their corresponding base models. It shows the improvement or degradation in short-context performance after long-context alignment. The baselines include models trained with supervised fine-tuning (SFT) and direct preference optimization (DPO). For reference, the improvements for GLM-9B-1M and LWM-7B-1M over their 128K counterparts (GLM-9B-128K and LWM-7B-128K) are also displayed. The MT-Bench metric, which originally ranges from 0 to 10, is linearly scaled to 0 to 100 for easier comparison across different tasks. Detailed numerical results can be found in Table 3.

read the caption

Figure 3: The margins of the short-context performance of Mistral-7B-LongPO and baselines relative to corresponding base model. GLM and LWM refer to the margins of GLM-9B-1M and LWM-7B-1M over GLM-9B-128K and LWM-7B-128K, respectively. MT-Bench metrics (∈\in∈[0, 10]) are linearly scaled to [0, 100] for better comparability across tasks. See numerical results in Table 3.

🔼 This figure compares the performance of LongPO with several baseline methods across both long- and short-context tasks. The baselines include: Supervised Fine-Tuning (SFT) using only the chosen responses, SFT using only the rejected responses, Direct Preference Optimization (DPO), and SFT with an added short-to-long constraint. The plots show the training progress over a number of steps, tracking performance on the RULER-NIAH (long-context) and MMLU (short-context) benchmarks. The figure illustrates LongPO’s ability to maintain strong performance on short-context tasks while simultaneously improving long-context performance, a key advantage over the baseline methods.

read the caption

Figure 4: Long- and short-context performance comparison among LongPO, SFT on chosen responses (SFT-Chosen), SFT on rejected responses (SFT-Rejected), DPO, and SFT on chosen responses with short-to-long constraint (SFT-Chosen-Constraint).

🔼 This figure shows the prompt used to instruct a large language model (LLM) to generate a pool of instructions for creating short-to-long preference data. The prompt guides the LLM to produce diverse, comprehensive questions that test a reader’s comprehension, analytical abilities, and the capacity to interconnect key themes across a document. These questions are designed to facilitate the creation of high-quality preference pairs by ensuring that each short-to-long comparison is meaningful and covers a broad range of the document’s content.

read the caption

Figure 5: The prompt for generating instruction pool.

🔼 This figure shows the reward trend for chosen responses during the training process of Mistral-7B-LongPO-128K. The reward, reflecting the model’s performance on selecting preferred responses, initially fluctuates before increasing steadily. This suggests the model effectively learns from the short-to-long preferences, improving its long-context capabilities over time.

read the caption

(a) The rewards for chosen response during training.

🔼 This figure shows the reward values for rejected responses during the training process of the Mistral-7B-LongPO-128K model. The rewards reflect the model’s preference for the chosen response over the rejected response as training progresses. Negative reward values indicate that the model prefers the chosen response. Observing the trend in the rewards for rejected responses can help assess the model’s alignment with the intended preferences and reveal if the model is learning to generate better long-context responses over the course of training.

read the caption

(b) The rewards for rejected response during training.

Full paper

#