Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond

↗ arXiv ↗ Hugging Face

TL;DR

#

Key Takeaways

#

Why does it matter?

#

The study offers a practical recipe for training COT models and highlights the importance of SFT data. This work opens doors for more cost-effective and accessible research in reasoning models, enabling researchers with limited resources to push the boundaries of AI.

Visual Insights

#

🔼 This figure illustrates the training pipeline used to develop the Light-R1-32B model. It shows the data processing steps, including data collection from various sources, decontamination to remove duplicates and inconsistencies, and difficulty filtering to select the most challenging problems for training. The model training phase is also depicted, highlighting the curriculum learning strategy with two stages of supervised fine-tuning (SFT) followed by direct preference optimization (DPO). This curriculum approach gradually increases the difficulty of the training data to improve model reasoning capabilities. The figure visually summarizes the complete workflow from raw data to the final Light-R1-32B model.

read the caption

Figure 1: Overview of training pipeline of Light-R1-32B.

🔼 This table presents an overview of the Light-R1 series of large language models. It shows the model name, performance metrics (AIME24, AIME25, GPQA, Diamond), and the training recipe used for each model. The ‘AIME’ scores are metrics for mathematical reasoning ability, while GPQA and Diamond measure performance on different reasoning tasks. The training recipes include stages of Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), with some models also undergoing Reinforcement Learning (RL) with GRPO. The ‘-DS’ suffix indicates models that were fine-tuned from DeepSeek-R1-Distill, while those without the suffix originated from Qwen-Instruct.

read the caption

Table 1: Light-R1 models. “-DS” = from DeepSeek-R1-Distill, otherwise from Qwen-Instruct.

In-depth insights

#

Long COT from RL

#

Reinforcement Learning (RL) for long Chain-of-Thought (CoT) presents a significant avenue for enhancing reasoning capabilities in language models. The core idea revolves around training models to generate more accurate and complete CoTs, leading to improved performance in complex tasks. The challenge lies in the complexity of RL, as it requires carefully designed reward functions to guide the model toward desired behaviors. A balance needs to be found between exploration and exploitation, ensuring that the model discovers novel reasoning paths while maintaining consistency and reliability. Additionally, the computational cost of RL can be substantial, necessitating efficient training algorithms and infrastructure. Despite these hurdles, the potential benefits of RL for long CoT are immense, paving the way for more robust and intelligent language models.

Curriculum Design

#

Curriculum design is critical for training long-context models, particularly from scratch. A multi-stage approach with increasing difficulty helps the model gradually acquire complex reasoning skills. The process starts with a general dataset that introduces fundamental concepts and then transitions to more challenging problems requiring sophisticated reasoning. This gradual exposure prevents the model from being overwhelmed and ensures effective learning. Data selection is also important which involves strategies for identifying and prioritizing valuable training examples.

SFT+DPO Scaling

#

Scaling SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization) presents a compelling paradigm for enhancing language model capabilities. The effectiveness of SFT hinges on the quality and diversity of the training data, while DPO leverages human preferences to align models with desired behaviors. Scaling both aspects requires careful consideration: For SFT, this involves expanding the dataset with high-quality examples that cover a wide range of tasks and reasoning abilities. DPO scaling involves gathering more preference data, potentially through active learning strategies that prioritize examples where the model is uncertain. Successfully scaling SFT+DPO needs balancing between model capacity, data quality, and computational costs, paving the way for significant advancements in language model performance and alignment.

R1 Reproducibility

#

The paper emphasizes the importance of reproducible results in the context of training large language models for long-chain-of-thought reasoning. It details the establishment of stable and trustworthy evaluation protocols mirroring those used by DeepSeek-AI to ensure the reliability of their findings. This commitment to reproducibility is evident in the release of evaluation code and logs, enabling others to independently verify the reported performance. The ability to reproduce results from other prominent models like DeepSeek-R1 and Qwen further strengthens the credibility of the research and provides a solid foundation for future work. The paper notes difficulties in matching the performance of models such as DeepSeek-R1 with smaller models trained on less data, highlighting the challenges in achieving comparable results even with open-source efforts. The study meticulously addresses data contamination issues, which is a crucial aspect of reproducibility, by rigorously cleaning training datasets to avoid biases and inaccurate performance evaluations. They thoroughly decontaminate datasets to ensure a fair comparison with benchmark datasets, which is a key factor of reproducibility. Reproducibility is central to validating their curriculum SFT, DPO, and RL approach for long COT model training.

High Data Quality

#

High data quality is paramount for training effective AI models, especially in tasks requiring reasoning and problem-solving. The quality of data directly impacts a model’s ability to learn, generalize, and perform well on unseen examples. Data quality encompasses various aspects, including accuracy, completeness, consistency, and relevance. Accurate data ensures the model learns from correct information, preventing the propagation of errors and biases. Complete data provides the model with all the necessary information to understand the context and make informed decisions. Consistent data ensures that the model can rely on the information it receives, without being confused by conflicting or contradictory inputs. Finally, relevant data ensures that the model focuses on the most important information for the task at hand. Maintaining high data quality requires careful data collection, cleaning, and validation processes. This may involve manual inspection, automated checks, and the use of data augmentation techniques to address data gaps and inconsistencies. Investing in high data quality is essential for building reliable and trustworthy AI systems.

More visual insights

#

Full paper

#