Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge

🔼 This figure illustrates the Crowd-based Comparative Evaluation (CCE) process. For a given task (x) and two candidate responses (yA and yB), the system first generates additional responses (crowd responses, yi) using an LLM. These crowd responses are then compared with the original candidate responses to create initial judgments (J). These judgments are refined (J^) through selection and processing steps that prioritize informative comparisons. Finally, the refined judgments are used as context for the LLM to produce a more comprehensive evaluation of the original candidates (yA, yB).

read the caption

Figure 2: Pipeline of our proposed crowd-based comparative evaluation. For a given instance (x,yA,yB)𝑥superscript𝑦𝐴superscript𝑦𝐵(x,y^{A},y^{B})( italic_x , italic_y start_POSTSUPERSCRIPT italic_A end_POSTSUPERSCRIPT , italic_y start_POSTSUPERSCRIPT italic_B end_POSTSUPERSCRIPT ), we first use the LLM to generate crowd responses {yi|i∈{C,D,E,…}}conditional-setsuperscript𝑦𝑖𝑖𝐶𝐷𝐸…\left\{y^{i}|i\in\{C,D,E,...\}\right\}{ italic_y start_POSTSUPERSCRIPT italic_i end_POSTSUPERSCRIPT | italic_i ∈ { italic_C , italic_D , italic_E , … } } based on x𝑥xitalic_x. These responses are then compared with yAsuperscript𝑦𝐴y^{A}italic_y start_POSTSUPERSCRIPT italic_A end_POSTSUPERSCRIPT and yBsuperscript𝑦𝐵y^{B}italic_y start_POSTSUPERSCRIPT italic_B end_POSTSUPERSCRIPT to produce initial crowd judgments 𝒥𝒥\mathcal{J}caligraphic_J, which are subsequently refined into 𝒥^^𝒥\hat{\mathcal{J}}over^ start_ARG caligraphic_J end_ARG after selection and processing. Finally, 𝒥^^𝒥\hat{\mathcal{J}}over^ start_ARG caligraphic_J end_ARG are used as contextual input to evaluate the instance (x,yA,yB)𝑥superscript𝑦𝐴superscript𝑦𝐵(x,y^{A},y^{B})( italic_x , italic_y start_POSTSUPERSCRIPT italic_A end_POSTSUPERSCRIPT , italic_y start_POSTSUPERSCRIPT italic_B end_POSTSUPERSCRIPT ).

🔼 This figure displays the impact of increasing the number of crowd judgments on the performance of the proposed Crowd-based Comparative Evaluation (CCE) method. The x-axis represents the number of crowd judgments included, while the y-axis shows two key metrics: accuracy of the LLM-as-a-judge and the length of the chain-of-thought (CoT) reasoning produced by the LLM. The graphs illustrate that as more crowd judgments are considered (increasing context), both the accuracy of the evaluation and the detail level of the CoT reasoning generally improve.

read the caption

Figure 3: Evaluation performance under scaling crowd judgments in the context. As the number of crowd judgments grows, both accuracy and CoT length generally increase.

🔼 Figure 4 presents a comparison of Chain-of-Thought (CoT) reasoning quality between the proposed Crowd-based Comparative Evaluation (CCE) method and the standard approach (Vanilla). It uses two key metrics: the average number of key points identified in the CoT and the coverage rate. The key points metric reflects how many distinct aspects or details the model considered when forming its judgment. The coverage rate indicates the proportion of relevant information in the input response that the model’s CoT addressed. Across all five benchmarks used for evaluation, CCE consistently outperformed the vanilla method in both of these metrics, demonstrating that CCE generates more comprehensive and in-depth CoTs.

read the caption

Figure 4: CoT Comparison. CCE’s CoT consistently yields a higher average number of key points and a higher coverage rate across all benchmarks.

🔼 This figure displays the prompt used in the Crowd-based Comparative Evaluation (CCE) method. The prompt instructs the LLM to act as an impartial judge, evaluating two AI-generated responses. It emphasizes holistic assessment, considering various criteria like helpfulness, relevance, accuracy, and completeness. The prompt also instructs the LLM to implicitly consider characteristics of other responses (crowd responses) as a background context, further refining its judgment. Crucially, the prompt explicitly forbids the LLM from referencing these background responses directly in its final verdict. The prompt concludes by specifying the format for the final verdict: ‘[[A]]’ or ‘[[B]]’, indicating which assistant performed better.

read the caption

Figure 5: Prompt of Our Method.

🔼 This figure displays the prompt used in the vanilla LLM-as-a-Judge method. Unlike the more complex prompt used in the Crowd Comparative Evaluation method (CCE), the vanilla prompt is simpler and focuses on the basic comparison of the two responses based on general criteria such as helpfulness, relevance, accuracy, depth, creativity, and level of detail. It instructs the LLM to choose the response that better answers the user’s question, emphasizing objectivity and avoiding bias based on factors such as length, style, order of presentation, or AI assistant names.

read the caption

Figure 6: Prompt of Vanilla LLM-as-a-Judge.

🔼 This figure displays the prompt used for the 16-criteria LLM-as-a-Judge evaluation method. The prompt instructs the LLM to act as an impartial judge, evaluating two AI-generated responses based on sixteen specified criteria, which are listed explicitly in the prompt, encompassing aspects such as helpfulness, relevance, accuracy, depth, creativity, detail level, overall quality, readability, coherence, fluency, grammaticality, simplicity, adequacy, faithfulness, non-hallucination and complexity. The LLM is directed to compare the responses, provide a short explanation, and render a final verdict indicating which response is superior, using a pre-defined format.

read the caption

Figure 7: Prompt of 16-Criteria LLM-as-a-Judge.

🔼 This figure shows the prompt used for the LongPrompt LLM-as-a-Judge baseline. The prompt instructs the LLM to act as an impartial judge, comparing two AI-generated responses to a given question. It emphasizes a detailed explanation of the comparison, aiming for a comprehensive evaluation of aspects like helpfulness, accuracy, depth, and creativity. The LLM is explicitly instructed to provide a long explanation. The final verdict must follow the format ‘[[A]]’ or ‘[[B]]’, indicating the better response.

read the caption

Figure 8: Prompt of LongPrompt LLM-as-a-Judge.

🔼 This table presents the accuracy results of smaller LLMs trained as judges using Chain-of-Thought (CoT) judgments generated by two different methods: Vanilla and Crowd-based Comparative Evaluation (CCE). The accuracy is measured across five pairwise comparison benchmarks (REWARDBENCH, HELPSTEER2, MTBENCH HUMAN, JUDGEBENCH, EVALBIAS). The table shows that using CCE to generate CoT judgments leads to significantly better performance of the trained judge models compared to using Vanilla. The highest accuracy scores for each benchmark are highlighted in bold, and the second-highest scores are underlined, demonstrating the consistent superior performance of the CCE approach.

read the caption

Table 2: Accuracy of Trained small LLM-as-a-Judge on pair-wise comparison benchmarks. Under the same preference pairs data, the model trained with judgments synthesized using CCE achieves more reliable evaluation results. The highest values are bolded, and the second highest is underlined.

🔼 This table presents the results of an experiment evaluating the effectiveness of different response sampling methods for supervised fine-tuning (SFT) within the context of instruction-following tasks. Three methods were compared: random sampling, vanilla rejection sampling (using a standard LLM-as-a-Judge), and crowd-based rejection sampling (using the proposed CCE method). The table shows the performance of models fine-tuned using each sampling method, measured on two instruction-following benchmarks (MTBENCH and ALPACAEVAL-2) with different numbers of training instructions (LIMA with 1K instructions, and TULU3-SFT with 10K instructions). The performance is assessed using two metrics, and the results demonstrate the improved generative capabilities of models trained with responses selected using the proposed CCE method.

read the caption

Table 3: SFT Rejection Sampling Performance on the Instruction-Following Benchmark. The model fine-tuned with responses sampled using CCE demonstrates improved generative performance.

🔼 This table presents the results of evaluating the performance of LLM-as-a-Judge models on several pairwise comparison benchmarks. It compares the accuracy of different methods, including the proposed Crowd-based Comparative Evaluation (CCE), majority voting (Maj@16), and criteria expansion (16-Criteria). The results show that CCE consistently improves the accuracy of LLM-as-a-Judge across various benchmarks, significantly outperforming both Maj@16 and 16-Criteria. Furthermore, it highlights that models trained using judgments generated by CCE achieve higher reliability in evaluation.

read the caption

Table 4: Accuracy of LLM-as-a-Judge on pair-wise comparison benchmarks. CCE can consistently enhance the LLM-as-a-Judge’s performance during the test-time inference phase, especially considerably outperforming maj@16 and 16-Criteria; under the same preference pairs data, the model trained with judgments synthesized using CCE achieves more reliable evaluation results.

🔼 Table 5 presents a case study comparing the evaluation results of a standard LLM-as-a-Judge approach and the proposed Crowd-based Comparative Evaluation (CCE) method. The example shows two model responses to a sentiment analysis task. The table highlights how CCE, by incorporating additional crowd responses and judgments, leads to a more nuanced and accurate evaluation compared to the standard LLM-as-a-Judge approach, which may be susceptible to noise or bias in individual judgments. Colors are used to distinguish between unreliable and helpful aspects of each evaluation. Limongreen highlights elements of evaluation considered useful or well-reasoned, while orange denotes unreliable or noisy aspects.

read the caption

Table 5: A pairwise comparison case evaluated by different methods. Preference refers to right result and Preference refers to wrong result. We emphasize the noisy evaluation elements in orange, while highlighting the useful elements of the evaluation in limongreen.

🔼 This table details the five preference benchmarks used to evaluate the LLM-as-a-Judge models. For each benchmark, it provides the number of instances (size) and a brief description of its focus, highlighting the types of scenarios or aspects it evaluates. These benchmarks encompass diverse evaluation scenarios, including chat, reasoning, and bias detection, to provide a comprehensive assessment of LLM-as-a-Judge performance.

read the caption

Table 6: The brief description of Preference Benchmarks for testing.