rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

🔼 This figure shows an example of a Code-augmented Chain of Thought (CoT) used in the rStar-Math model. The example problem is a word problem asking to calculate the direct distance from a starting point after a series of movements. The solution is presented step-by-step using a code-augmented CoT, where each step involves natural language reasoning (the NL CoT) accompanied by executable Python code. The Python code directly implements the reasoning described in the accompanying natural language. This approach ensures the correctness and verifiability of each step, mitigating the problem of hallucination often seen in large language models (LLMs) that generate only natural language reasoning steps.

read the caption

Figure 2: An example of Code-augmented CoT.

🔼 This figure displays the impact of increasing computational resources during testing on the accuracy of mathematical problem-solving. Four different benchmarks (MATH, AIME 2024, Olympiad Bench, and College Math) are shown, each represented by a separate graph. The x-axis represents the number of solutions sampled during the test-time computation, and the y-axis shows the percentage accuracy achieved. Multiple models are compared: OpenAI’s o1-preview and o1-mini models, and the rStar-Math model with a 7B parameter policy LLM and a 7B parameter PPM, as well as a comparison using Qwen2.5 Best-of-N models with 7B and 72B parameter LLMs and a 72B parameter ORM. This allows for a visual comparison of how different models and levels of computational scaling (number of solution samples) impact the accuracy of mathematical reasoning across the various benchmarks. The plot shows that increasing the number of samples generally improves accuracy, but the rate of improvement varies for different models and benchmarks. It also highlights the competitive performance of rStar-Math, sometimes exceeding even the larger models with less compute.

read the caption

Figure 3: Reasoning performance under scaling up the test-time compute.

🔼 This figure shows an example of the model’s intrinsic self-reflection ability during problem-solving. The model initially attempts a solution using SymPy, but realizes it’s leading to an incorrect answer (as indicated by the low PPM score). It then abandons this approach, and instead finds a simpler and correct solution. This demonstrates the model’s ability to identify and correct its own mistakes, rather than simply continuing down an incorrect path.

read the caption

Figure 4: An example of intrinsic self-reflection during \sysname deep thinking.

🔼 Figure 5 presents a comparative analysis of the performance of various policy models (different sizes) before and after integrating them into a System 2 reasoning framework. The x-axis represents different mathematical benchmarks (MATH, AIME 2024, AMC 2023, Olympiad Bench, College Math), while the y-axis shows Pass@1 accuracy. The bars visually represent the accuracy of individual policy models (various base models) using only System 1 reasoning (i.e., generating a single answer without deep thinking). The bars also show the improved performance achieved by incorporating these models into a System 2 deep thinking approach with various reward models (ORM, PPM). The key takeaway is that the reward model significantly impacts the final performance of the system, highlighting its importance in System 2 deep thinking.

read the caption

Figure 5: Pass@1 accuracy of policy models and their accuracy after applying System 2 reasoning with various reward models, shows that reward models primarily determine the final performance.

🔼 Figure 6 presents a comparison of the accuracy (Pass@N) achieved using random sampling for solution selection from various policy models. The x-axis represents the number of solutions sampled (N), while the y-axis shows the accuracy. Four different benchmarks are shown: MATH, AIME 2024, Olympiad Bench, and College Math. The results demonstrate that the rStar-Math policy models consistently outperform the official Qwen instruct versions, showcasing their superior ability to generate correct solutions, even when only a small number of samples are considered.

read the caption

Figure 6: Pass@N accuracy with random sampling from different policy models. Compared to the official Qwen instruct version, our policy model exhibits a stronger ability to sample correct solutions.

🔼 Figure 7 illustrates the performance of four different sized language models (1.5B, 3.8B, 7B, and 72B parameters) on several math reasoning benchmarks when using Monte Carlo Tree Search (MCTS) with a process preference model (PPM). The key takeaway is that despite the varying sizes of the base models, their accuracy in sampling correct solutions converges as the number of MCTS samples increases. This highlights the effectiveness of the PPM in guiding the search process, regardless of the underlying model’s size.

read the caption

Figure 7: Pass@N accuracy with PPM-augmented MCTS. Under the same PPM guidance, the four policy models of varying sizes demonstrate convergent capabilities in sampling correct solutions.

🔼 This table shows the percentage of 747,000 math problems that were correctly solved by the model in each of four training rounds. Only problems with verifiable correct solutions were included in the training data. The first round used a pre-trained large language model (DeepSeek-Coder-Instruct) as the initial policy model. Subsequent rounds used a smaller, fine-tuned 7-billion parameter language model that was iteratively improved during the self-evolution process. The table provides a measure of the model’s progress in solving increasingly difficult math problems across the training rounds.

read the caption

Table 2: Percentage of the 747k math problems correctly solved in each round. Only problems have correct solutions are included in the training set. The first round uses DeepSeek-Coder-Instruct as the policy LLM, while later rounds use our fine-tuned 7B policy SLM.

🔼 Table 3 presents the accuracy (Pass@1) of the policy model across four rounds of training. The initial policy model (Round 0, bootstrap model) is a pre-trained model. Each subsequent round represents an iterative improvement of the policy model, trained using data generated by the previous round. The table demonstrates consistent accuracy improvement across rounds, culminating in performance that surpasses that of the initial, pre-trained bootstrap model.

read the caption

Table 3: Pass@1 accuracy of the resulting policy SLM in each round, showing continuous improvement until surpassing the bootstrap model.

🔼 Table 4 presents the performance of the Process Preference Model (PPM) across four rounds of training. The key point is that the PPM’s ability to accurately evaluate the quality of reasoning steps improves with each round. To ensure a fair comparison and isolate the PPM’s progress, the policy model used remained consistent (policy SLM-r1) throughout the four rounds. The table shows accuracy scores across various math reasoning benchmarks (MATH, AIME 2024, AMC 2023, Olympiad Bench, College Math, GSM8K, GaokaoEn 2023) for each round’s PPM, highlighting the steady improvement in performance.

read the caption

Table 4: The quality of PPM consistently improves across rounds. The policy model has been fixed with policy SLM-r1 for a fair comparison.

🔼 Table 5 presents a comparison of the performance of the rStar-Math model against state-of-the-art Large Language Models (LLMs) on a variety of challenging mathematical benchmarks. It shows the Pass@1 accuracy (the percentage of problems solved correctly in a single attempt) for each model across different datasets, including MATH, AIME 2024, AMC 2023, Olympiad Bench, College Math, GSM8K, and GaokaoEn 2023. The table also includes results for rStar-Math64, which represents the performance when the model generates 64 solution attempts for each problem, showcasing the impact of increased computational resources on accuracy. The benchmarks range in difficulty, from commonly used datasets like GSM8K to more challenging competition-level datasets such as AIME (American Invitational Mathematics Examination) and Olympiad Bench, providing a comprehensive evaluation of the models’ capabilities.

read the caption

Table 5: The results of \sysname and other frontier LLMs on the most challenging math benchmarks. \sysname64 shows the Pass@1 accuracy achieved when sampling 64 trajectories.

🔼 This table presents the performance of the rStar-Math model across four rounds of self-evolution. It shows how the model’s accuracy improves on various mathematical reasoning benchmarks (MATH, AIME 2024, AMC 2023, Olympiad Bench, College Math, GSM8K, GaokaoEn 2023) as it iteratively learns and refines its reasoning capabilities. Notably, starting from round 2, the 7B base model enhanced by rStar-Math surpasses the performance of GPT-4, a leading large language model, demonstrating the effectiveness of the self-evolution approach.

read the caption

Table 6: The continuously improved math reasoning capabilities through \sysname self-evolved deep thinking. Starting from round 2, the 7B base model powered by \sysname surpasses GPT-4o.

🔼 This table presents an ablation study evaluating the effectiveness of using step-by-step verified reasoning trajectories as a training dataset for fine-tuning the Qwen2.5-Math-7B language model. It compares the model’s performance after fine-tuning on this dataset against its performance after training on several other datasets, including those generated by GPT-4 and other methods such as random sampling and rejection sampling. The results are reported as Pass@1 accuracy across various mathematical reasoning benchmarks, allowing for a comparison of the different training approaches and their impact on the final model’s accuracy.

read the caption

Table 7: Ablation study on the effectiveness of our step-by-step verified reasoning trajectories as the SFT dataset. We report the SFT accuracy of Qwen2.5-Math-7B fine-tuned with different datasets.

🔼 This table presents an ablation study comparing three different reward model approaches for a System 2 math reasoning model: an Outcome Reward Model (ORM), a Process Reward Model using Q-values (PQM), and a Process Preference Model (PPM). The goal is to assess the impact of the reward model on the overall accuracy of the system. The results show that both PQM and PPM outperform the ORM, demonstrating the benefit of using fine-grained feedback on the reasoning process rather than only considering the final outcome. The PPM achieved the best performance and enabled the model to reach state-of-the-art accuracy, surpassing even the ORM used with a much larger model.

read the caption

Table 8: Ablation study on the reward model. Process reward models (PQM and PPM) outperform ORM, with PPM pushing the frontier of math reasoning capabilities.

🔼 This table presents the average number of tokens generated by the rStar-Math model to produce a single reasoning trajectory for different types of math problems. The token counts reflect the computational cost associated with generating solutions using Monte Carlo Tree Search (MCTS) for various problem difficulty levels (MATH, AIME 2024, AMC 2023, Olympiad Bench, College Math, GSM8K, GaokaoEn 2023). These costs are important for understanding the efficiency and scalability of the proposed method.

read the caption

Table 9: Inference costs of \sysname. We show the average number of generated tokens required to generate a trajectory for a given question.

🔼 This table presents the Pass@1 accuracy, a measure of how often the model’s top prediction is correct, for four different sized language models after fine-tuning. The models are evaluated on several math reasoning benchmarks, including MATH, AIME 2024, AMC 2023, Olympiad Bench, College Math, GSM8K, and GaokaoEn 2023. It compares the performance of the base, instruct-tuned, and rStar-Math-fine-tuned versions of each model to demonstrate the impact of the proposed method.

read the caption

Table 10: Pass@1 (greedy) accuracy of our fine-tuned policy models for Phi3-mini, Qwen2.5-Math-1.5B, Qwen2-Math-7B and Qwen2.5-Math-7B.