WILDCHAT-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training

🔼 This figure shows the impact of increasing the size of the training dataset on the performance of Supervised Fine-Tuning (SFT). The x-axis represents the size of the training dataset, while the y-axis shows the average performance across various benchmarks (MixEval, AlpacaEval2-LC, MTBench/10, and OpenLLM LB2). Different colored bars represent different Data Generating Models (DGMs). The results indicate that larger datasets generally lead to improved SFT performance; however, the degree of improvement varies depending on the quality of the synthetic data generated by the DGM. For models like GPT-3.5, performance plateaus relatively quickly with increased data size, while other DGMs show consistent improvement with larger datasets. This highlights that data quality and the model used for data generation significantly influence the effectiveness of data scaling in SFT.

read the caption

Figure 2: Data scaling improves SFT performance. The effect is, however, somewhat dependent on SDQ – for DGMs such as GPT 3.5, the benefits taper off relatively quickly, but for the other three DGMs we consider, they continue to increase. Avg is the average performance over (MixEval, AlpacaEval2-LC, MTBench / 10, OpenLLM LB 2).

🔼 Figure 3 is a word cloud showing the words that appeared most frequently in the human evaluator’s feedback when comparing the performance of two fine-tuned language models (LLMs): L8B:L70 (Llama-3.1 8B Base fine-tuned on Llama-3.3-70B responses) and L8B:Q72 (Llama-3.1 8B Base fine-tuned on Qwen-2.5-72B responses). The word cloud visualizes the negative sentiment expressed in the evaluations of L8B:L70. The prominent words, such as ’lacks,’ ‘convoluted,’ and ‘repetitive,’ highlight the perceived shortcomings of L8B:L70 in comparison to L8B:Q72. The presence of ‘clearer’ indicates that the judges desired more clarity in L8B:L70’s outputs. The word cloud provides a concise visual summary of the qualitative aspects of model performance.

read the caption

Figure 3: Key words more common in L8B : L70 judgments. The more negative tone of these judgments emphasizes words like clearer (as in, “could have been clearer”), lacks, convoluted and repetitive.

🔼 Figure 4 is a word cloud visualizing the words that frequently appeared in the LLM judge’s feedback when comparing the performance of Llama-3.1-8B-Base fine-tuned on Qwen-2.5-72B responses against other models. The word cloud shows that the judges’ comments were generally more positive for this model. Words like ‘appropriate,’ ’necessary,’ ‘comprehensive,’ and ‘accurate’ were prominently featured, indicating that the model’s responses were perceived as well-suited to the task and of high quality.

read the caption

Figure 4: Key words more common in L8B : Q72 judgments. These judgments tended to be more positive; emphasis was placed on words like appropriate, necessary, comprehensive and accurate.

🔼 This table presents a comparative analysis of six different large language models (LLMs) used as data generators for supervised fine-tuning (SFT). Each LLM, ranging in size from 0.5 billion to 104 billion parameters, generated 250,000 chat transcripts. The performance of the fine-tuned models is then evaluated across various benchmarks, revealing that the choice of data-generating model has a significant and unpredictable impact on the final performance. There is a considerable amount of variance in the results, with no single model consistently outperforming the others.

read the caption

Table 2: The choice of data generating model has strong and unpredictable effects on downstream benchmark performance. We compare the performance of six different DGMs from four different model families, ranging in size from 0.5B to 104B parameters, each fine-tuned on 250k samples from a DGM. We find a large degree of variance in benchmark performance, with no one model dominating.

🔼 Table 3 presents a detailed analysis of stylistic element inheritance during supervised fine-tuning (SFT) of large language models (LLMs). It compares the frequency of specific Markdown formatting elements (converted to HTML for clarity) in the outputs of both original data-generating models (DGMs) and their corresponding SFT models. The table shows the absolute frequency counts of various HTML tags and inline properties in 80 turns from the MTBench dataset, as well as the proportional frequency of these elements between the DGM and its SFT counterpart. The ‘proportional frequency’ metric helps assess how closely the SFT model’s style matches that of the DGM it was trained on, enabling researchers to understand the extent of style transfer during SFT and its potential impact on downstream model performance. Note that the order of the models in computing the proportional frequency is arbitrary.

read the caption

Table 3: SFT models strongly inherit formal stylistic elements from their DGMs. This table indicates how frequently certain Markdown stylistic elements appear in LLM responses (converted to HTML tags for greater clarity). The columns are names of HTML tags and inline properties, and the cells are frequency counts. PF stands for proportional frequency, the ratio of the first and second model listed (order here is presumed to be arbitrary).

🔼 This table presents the results of a comparative analysis on the performance of different Data Generating Models (DGMs) and their corresponding Supervised Fine-Tuning (SFT) models. It evaluates how often LLMs prefer the responses generated by each DGM and SFT model, using various benchmarks. The ‘Preference Rate’ columns show the percentage of times one model’s response was preferred over another across different evaluation metrics (Benchmarks). The ‘FlipCt’ column represents the number of instances where the ranking between the DGM and SFT model is reversed in terms of preference. The analysis highlights the strong preference for Qwen responses among LLMs and the consistent preference pattern observed between DGMs and SFT models, demonstrating the impact of the original DGM’s choice on the SFT model performance.

read the caption

Table 4: LLM judge preferences for DGMs and SFTs. For column name definitions, we refer the reader to the main text, section Sec. 3. Overall, we observe that Qwen responses are generally preferred by LLM judges, and that the rate at which they are preferred is similar from DGM to SFT. Last but not least, we note that reversals of judgment from SFT to DGM (FlipCt) are uncommon.

🔼 This table presents the results of an intra-LLM response similarity analysis conducted on the WILDCHAT-50M dataset. It shows how similar the responses generated by different large language models (LLMs) are to each other, using three common Natural Language Processing (NLP) metrics: ROUGE-1, ROUGE-L, and METEOR. The metrics measure the overlap between the responses. Higher scores indicate greater similarity. The table shows that the models exhibit a relatively high degree of response similarity. The methodology used to calculate these scores is described in detail in Section 2.2 of the paper. This similarity across models is noteworthy, given the diversity of prompts and the range of models used in the study. It suggests that LLMs, regardless of their size or training, tend to generate responses that are more similar to each other than one might expect based on human responses to the same prompts. This observation hints at the relatively limited diversity found in LLMs’ responses.

read the caption

Table 5: Intra-LLM response similarity in Re-Wild. Here we report intra-llm response similarity scores. The method we use to obtain these scores is described in Sec. 2.2.

🔼 Table 6 presents a comprehensive analysis of the impact of using similar data generating models (DGMs) during supervised fine-tuning (SFT) on the performance of large language models (LLMs). It compares different combinations of base LLMs (Llama and Qwen) and fine-tuned models, each trained on datasets generated by various DGMs. The table shows that fine-tuning an LLM on data from a similar DGM leads to significantly better performance across various benchmarks compared to using data from dissimilar DGMs. This highlights the importance of DGM selection for effective SFT and indicates that models might learn more effectively when the training data reflects their own characteristics and style.

read the caption

Table 6: Models learn more effectively from highly similar DGMs. In this table, we report the complete, extended results from our experiments on the effect of diversifying both DGM and SFT-target. Both Llama and Qwen benefit from more similar upstream models.

🔼 This table presents the results of experiments assessing the impact of blending different Data Generating Models (DGMs) on the performance of downstream models. The experiments involved fine-tuning models on datasets created by mixing data from different DGMs in roughly equal proportions. The table shows the performance of these models on various benchmark tasks (MTBench, AlpacaEval, BBH, GPQA, MATH, MUSR, IFEval, MMLU, Pro, and MixEval). The ‘Avg’ column shows the average performance across all benchmarks. The results indicate that blending DGMs generally leads to performance that falls between the performance of the individual DGMs used in the mixture, rather than surpassing the performance of the best individual DGM.

read the caption

Table 7: Effects of blending DGMs. Where we blend models, we always draw a random sample of approximately equal size from each DGM, and denote the mixture as DGM A + DGM B. e.g., L8I + Q7. In this table, it can be seen that blends benchmark between their DGMs, not above them (as measured by Avg).