Free Process Rewards without Process Labels

↗ arXiv ↗ Hugging Face ↗ Papers with Code

TL;DR

#

Key Takeaways

#

Why does it matter?

#

This paper is crucial because it significantly reduces the cost and effort of training process reward models (PRMs), a critical advancement for various AI applications. Its novel approach makes PRM training accessible to a broader research community and enables further investigation into more complex AI tasks. The findings challenge existing assumptions about PRM training, opening up new avenues of research and potential breakthroughs in areas such as reinforcement learning and decision-making. The method is particularly beneficial for scenarios with scarce data, making high-quality PRMs accessible for a wider range of tasks and datasets.

Visual Insights

#

🔼 This figure compares different reward models based on their training cost (FLOPs) and performance (best-of-64 accuracy on the MATH dataset). The performance is averaged across three different language models: Mistral-7B-Instruct-v0.2, Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct. Each point on a line represents a model trained with the same method but with a different amount of training data. The ideal model would be located in the top-left corner, indicating high accuracy with low training cost. The results show that the proposed ‘implicit PRM’ achieves the best accuracy while having significantly lower training costs compared to other methods.

read the caption

Figure 1: The x-axis indicates the FLOPs required to collect the data and train the model, and y axis the accuracies of best-of-64 performance. The accuracy is averaged over the best-of-64 accuracies of Mistral-7B-Instruct-v0.2 (Jiang et al., 2023), Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct (Meta, 2024) on MATH (Hendrycks et al., 2021). Different dots on the same line indicates models trained with the same approach but on different scales of data. The top-left zone is desirable in this figure, as it suggests a model can achieve higher performance with less development overhead. Our implicit PRM is much cheaper to train while presenting the best performance under the same budget.

🔼 This table presents a comparison of various reward models’ performance on the MATH dataset, using best-of-N sampling. Three different large language models (LLMs) were used as generation models: Mistral-7B-Inst-v0.2, Llama-3.1-8B-Inst, and Llama-3.1-70B-Inst. The models’ individual pass@1 accuracies (when completing instructions with a temperature of 0.5) are shown in the caption (9.6%, 44.6%, and 63.2% respectively). The table shows the performance (pass@k, where k = 4, 16, and 64) of different reward models, categorized as outcome reward models (ORMs) and process reward models (PRMs), including both open-source and custom-implemented models. The goal is to evaluate the effectiveness of various reward models in improving the accuracy of the LLMs during best-of-N sampling.

read the caption

Table 1: Different reward models’ best-of-N sampling performance on MATH test set with three different generation models. When completing instructions with a temperature of 0.5, the three generation models’ accuracies are 9.6%, 44.6%, and 63.2% respectively.

In-depth insights

#

Implicit PRM Training

#

Implicit PRM training offers a groundbreaking approach to process reward model (PRM) development by eliminating the need for costly and time-consuming step-level annotations. Instead of directly training a PRM on meticulously labeled intermediate steps, this method cleverly leverages the readily available response-level labels from outcome reward model (ORM) training. The core idea is to parameterize the reward function as the log-likelihood ratio between a policy model and a reference model. This clever technique allows the ORM to implicitly learn a PRM during its training phase. This paradigm shift is particularly significant because it dramatically reduces the data requirements and computational burden associated with PRM training, making the technology significantly more accessible. The implied PRM’s performance is surprisingly competitive with, and even surpasses, explicitly trained PRMs that utilize computationally expensive methods like Monte Carlo Tree Search (MCTS) for data augmentation. Furthermore, this approach’s inherent flexibility makes it adaptable to diverse loss functions, further enhancing its versatility and practicality. This innovative approach represents a major advancement in PRM training, promising to democratize access to this powerful technique for various applications.

Reward Parameterization

#

Reward parameterization is a crucial aspect of reward modeling, particularly in the context of process reward models (PRMs). The choice of parameterization significantly impacts the model’s ability to learn and generalize effectively. A well-chosen parameterization can simplify the training process and lead to better performance, potentially avoiding the need for expensive step-level labels in the case of PRMs. This paper explores parameterizing the reward as the log-likelihood ratio of policy and reference models. This approach offers several advantages: it implicitly obtains a PRM at no additional cost, it’s objective-agnostic, applicable to various loss functions (e.g., DPO and CE), and theoretically avoids the noisy estimations associated with methods like MCTS. The choice of reference model is another key consideration, potentially impacting efficiency and performance. While the reference model can add inference cost, the authors demonstrate scenarios where it can be omitted without significant performance loss. Finally, the study of different reward parameterization choices (e.g., linear transformations of hidden states, generative logits) helps highlight the advantages of log-likelihood ratio parameterization for balancing efficiency and performance in PRM training.

Data Efficiency Gains

#

The concept of ‘Data Efficiency Gains’ in the context of process reward models (PRMs) centers on reducing the substantial data requirements associated with training these models compared to outcome reward models (ORMs). Traditional PRM training demands labels for every intermediate step of a reasoning process, leading to significant annotation costs and limitations in scalability. The research likely explores methods to achieve comparable or improved performance with fewer labeled steps, perhaps by leveraging the information implicitly present in the ORM training data. This could involve sophisticated reward parameterization techniques, as suggested in some research, which enable the extraction of process-level feedback from full response-level feedback, effectively utilizing data more efficiently. The analysis would involve assessing the performance improvements achieved with reduced data under various objectives, evaluating the impact of different loss functions, and perhaps comparing against strong MCTS-based baselines. Ultimately, the goal is to demonstrate substantial data efficiency improvements while maintaining or exceeding the performance of conventional PRM training approaches, making PRM training more accessible and scalable for broader applications.

Majority Voting Boost

#

The concept of a ‘Majority Voting Boost’ in the context of a process reward model (PRM) for enhancing large language model (LLM) performance is intriguing. It suggests that combining the individual predictions of multiple PRMs, possibly trained with different objectives or data subsets, and selecting the final prediction based on a majority vote could significantly improve accuracy. This approach leverages the collective wisdom of multiple models to mitigate individual weaknesses and biases. The effectiveness hinges on the diversity of the PRMs; if they all make similar mistakes, majority voting won’t help. Therefore, diverse training data, model architectures, or objective functions are crucial to realize the potential of this boost. An interesting point to explore would be the specific voting strategy employed: simple majority, weighted majority based on individual model performance, or more sophisticated ensemble methods. A further consideration is the computational cost associated with training and running multiple PRMs, and whether this increase is justified by the improvement in accuracy. The potential to optimize computational resources while maintaining the accuracy benefit of majority voting should also be investigated. Finally, a detailed analysis of the types of errors each PRM makes and how majority voting corrects these errors would provide a valuable understanding of this technique’s efficacy.

Limitations of PRMs

#

Process Reward Models (PRMs), while offering advantages over Outcome Reward Models (ORMs) by providing finer-grained feedback, suffer from significant limitations. Data acquisition for PRMs is substantially more expensive, requiring annotations for each intermediate reasoning step, unlike ORMs which only need final outcome labels. This makes large-scale PRM training challenging. The accuracy of automatically generated step-level labels, often relying on methods like Monte Carlo Tree Search (MCTS), is questionable; noisy labels can negatively impact PRM performance. Furthermore, the effectiveness of PRMs is heavily dependent on the quality of the underlying language model used for generation, raising concerns about potential bias amplification and the difficulty in achieving reliable and consistent performance across diverse tasks. Finally, the increased computational cost during both training and inference due to the handling of multiple steps presents a practical hurdle for widespread adoption. Addressing these limitations is crucial for wider application of PRMs in complex reasoning tasks.

More visual insights

#

More on figures

🔼 This figure illustrates the computational cost of training different process reward models (PRMs), comparing the proposed implicit PRM with existing methods. The x-axis represents the number of responses per instruction, which directly impacts the size of the training dataset. The y-axis displays the total FLOPs (floating-point operations) required for both data collection and model training. The figure demonstrates that the implicit PRM consistently requires significantly fewer FLOPs than other PRMs, highlighting its computational efficiency. Specifically, the implicit PRM using the cross-entropy (CE) loss function is shown to be 38.6 to 38.8 times more efficient than the Math-Shepherd method, showcasing a substantial reduction in computational resources.

read the caption

Figure 2: Overhead of developing different PRMs, in terms of FLOPs during data collection and training. The X axis indicates the number of responses per instruction which determines the scale of training data, and the Y axis is the number of FLOPs. Our implicit PRM always consumes the least FLOPs compared to baselines, with CE being 38.6×\times× to 38.8×\times× more efficient than Math-Shepherd across different dataset scales.

🔼 Figure 3 presents the average best-of-N accuracy across three different test sets (Mistral-7B-Instruct-v0.2, Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct) when using majority voting. The bar chart displays the performance improvement obtained by using majority voting to aggregate the scores of multiple model responses that lead to the same answer, compared to not using majority voting.

read the caption

Figure 3: Results with majority voting. We present the averaged best-of-N accuracy across three testsets.

🔼 The figure shows the results of scaling up the number of instructions used to train an implicit PRM using the DPO objective. The x-axis represents the number of candidates considered in the best-of-N sampling, while the y-axis shows the accuracy. Different colored lines represent the performance of the implicit PRM trained on different proportions of the original instruction dataset (25%, 50%, 75%, and 100%). The figure displays the results on three different generation models: Mistral-7B-Inst-v0.2, Llama-3.1-8B-Inst, and Llama-3.1-70B-Inst. It demonstrates how the performance of the implicit PRM scales with the amount of training data.

read the caption

(a) Implicit PRM (DPO).

🔼 This figure shows the results of scaling the number of responses per instruction when training an Implicit PRM using the cross-entropy (CE) loss. It presents the best-of-N sampling performance on three different language models (Mistral-7B-Inst-v0.2, Llama-3.1-8B-Inst, Llama-3.1-70B-Inst) across various numbers of responses per instruction (1, 2, 4, 8). The x-axis represents the number of candidates considered in best-of-N sampling, and the y-axis represents the accuracy. The figure illustrates how the model’s performance improves with more responses per instruction, showcasing the effect of data scaling on the Implicit PRM trained with CE loss.

read the caption

(b) Implicit PRM (CE).

🔼 This figure displays the results of experiments on scaling the number of instructions used to train implicit PRMs. The x-axis represents the number of candidates considered in the best-of-N sampling process. The y-axis represents the accuracy achieved. Three different language models (Mistral-7B-Instruct-v0.2, Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct) were used. The results show that for Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct, the performance of the implicit PRM generally improves as the number of training instructions increases. However, the trend is more complex and less consistent for Llama-3.1-70B-Instruct, highlighting potential differences in how these models benefit from increased training data.

read the caption

Figure 4: Scaling instruction numbers. Our implicit PRM’s performance on Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct scales well with the number of instructions, despite the trend is more complex on Llama-3.1-70B-Instruct.

🔼 The figure shows the result of scaling up the number of instructions for the Implicit PRM using the DPO objective. It presents the best-of-N sampling performance on three different language models: Mistral-7B-Inst-v0.2, Llama-3.1-8B-Inst, and Llama-3.1-70B-Inst. The x-axis shows the number of candidates considered in the best-of-N sampling, and the y-axis shows the accuracy. Different colored lines represent different scales of training data (25%, 50%, 75%, and 100% of the instructions). The figure demonstrates the effect of scaling instruction data on the performance of the Implicit PRM with DPO.

read the caption

(a) Implicit PRM (DPO).

Full paper

#