KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding

🔼 This figure visualizes the success rate of the self-verification process in Step 2 of the KODCODE dataset generation pipeline. It displays the pass rate for each subset of coding questions (Prefill, LeetCode, Codeforces, etc.) with varying numbers of self-verification attempts (1, 5, and 10). Higher pass rates indicate that the questions in that subset were generally easier to solve and verify, while lower pass rates may reflect increased difficulty. The graph allows assessment of the effectiveness of increasing verification attempts in generating verifiable solutions, showing how many more successful solutions are produced with more attempts.

read the caption

Figure 2: Statistics on pass rates via self-verification in Step 2 by subset with varing number of attempts.

🔼 This figure presents three distinct histograms visualizing the distribution of token lengths for questions and their corresponding responses across various subsets of the KODCODE dataset. A supplementary bar chart displays the average number of unit tests associated with each question in these subsets. This allows for a comparative analysis of the textual length characteristics of questions and answers, and the associated test suite complexity across different question categories within the dataset. This analysis helps to understand the diversity and challenge level within the dataset.

read the caption

Figure 3: Distribution of token lengths for questions and responses, along with unit test counts across different subsets.

🔼 This figure uses t-distributed Stochastic Neighbor Embedding (t-SNE) to visualize the high-dimensional vector representations of coding instructions from KODCODE and four other datasets: OSS Instruct, ACECoder, Educational Instruct, and Package Instruct. Each point represents a coding instruction, and the proximity of points indicates semantic similarity. The visualization reveals KODCODE’s superior diversity compared to the baseline datasets, as its instructions span a wider range of the embedding space. This indicates that KODCODE covers a broader range of coding topics and styles than the other datasets.

read the caption

Figure 4: Comparison of t-SNE visualization between KodCode  (by subset) and baseline datasets (OSS Instruct, ACECoder, Educational Instruct, and Package Instruct), with 2,000 sampled instructions per dataset.

🔼 This figure shows the distribution of question difficulty across different subsets of the KODCODE dataset. The difficulty is measured by the pass rate of the self-verification process for each question, using the number of successful attempts out of a maximum of 10. The subsets are categorized into four difficulty levels: easy (pass rate > 2/3), medium (1/3 to 2/3), hard (<1/3), and fail (all failures). The chart displays the percentage of questions in each difficulty level for every subset, providing a visual representation of the distribution of question difficulty in the dataset.

read the caption

Figure 5: Difficulty distribution across subsets measured by pass rates.

🔼 Figure 6 presents a detailed analysis of potential overlaps between the KODCODE dataset and existing code benchmarks. Part (a) is a confusion matrix that visually represents the number of KODCODE samples exhibiting high similarity (cosine similarity > 0.95) to each benchmark. The percentages and raw counts are provided within the matrix cells, illustrating the extent of potential contamination for each KODCODE subset. Part (b) provides a supplementary histogram showing the distribution of maximum cosine similarity scores within each KODCODE subset. Horizontal lines on the histogram highlight the average maximum similarity for each subset, offering a clearer picture of the overall similarity levels.

read the caption

Figure 6: Contamination analysis between KodCode  subsets and existing benchmarks. (a) Confusion matrix showing the percentage and absolute number (in parentheses) of contaminated samples with cosine similarity > 0.95. (b) Distribution of maximum cosine similarity scores across different KodCode subsets, with horizontal lines indicating subset averages.

🔼 This figure illustrates the data flow within the KODCODE dataset creation pipeline. Starting from twelve initial subsets of coding questions (left), the pipeline proceeds through a deduplication step to remove redundant questions (middle). The remaining questions then undergo a reject sampling process where solutions and tests are generated, and those that fail verification are removed (right). The red paths highlight questions removed during the deduplication and reject sampling phases. The final dataset is considerably smaller than the initial number of coding questions indicating the effectiveness of the data cleaning steps.

read the caption

Figure 7: Data flow visualization through our pipeline: from initial subsets (left) through deduplication (middle) to final filtered sets after reject sampling (right). Red paths indicate discarded instances.

🔼 This figure shows a pie chart illustrating the distribution of various coding task categories within the Magpie Coding dataset before any filtering was applied. The pie chart visually represents the proportion of questions in each category, providing an overview of the dataset’s initial composition before any refinement or selection processes were implemented. This helps illustrate the diversity of tasks addressed in the original Magpie data and the impact that subsequent filtering steps had on the final dataset composition.

read the caption

Figure 8: Task categories of the collected Magpie Coding data before filtering.

🔼 This figure shows a coding task (Task 511) from the MBPP dataset that failed the self-verification process during the KODCODE dataset generation. The task requires writing a Python function to find the minimum sum of factors for a given number. The figure displays the GPT-40 generated response (incorrect), the correct solution, and the associated unit tests.

read the caption

Figure 9: Failed Task 511.

🔼 This figure shows a failed example from the MBPP validation dataset (Task 525). The task is to write a Python function to determine if two lines are parallel. The function should accept two lines represented as tuples, handling both two-element (a,b) and three-element (a,b,c) tuples. The GPT-40 response’s solution incorrectly assumes all input lines are represented by three-element tuples (a,b,c), resulting in a failed unit test when only two-element tuples are provided. This highlights a limitation of the GPT-40 model in handling edge cases and generalizing to different input formats.

read the caption

Figure 10: Failed Task 525.

🔼 This figure shows an example of data contamination found in the KODCODE dataset. It highlights a question from KODCODE that is nearly identical to a question from the MBPP benchmark dataset. Specifically, it demonstrates a KODCODE question about counting uppercase letters in a string and its almost exact equivalent from MBPP. This illustrates a potential limitation in the dataset generation process, where some synthetically generated questions closely resemble pre-existing questions from other established benchmarks.

read the caption

Figure 11: Example 1 of Contaminated Data

🔼 Table 2 presents a comparison of the performance of various large language models (LLMs) on five different coding benchmarks: HumanEval, MBPP, BigCodeBench-Complete, BigCodeBench-Instruct, and LiveCodeBench. The models compared include several strong baseline LLMs and two models fine-tuned using the KodCode dataset. For each benchmark, sub-metrics were averaged, and then the average across all five benchmarks is reported. Bold values indicate the highest score for each metric. The results demonstrate that LLMs fine-tuned on the KodCode dataset significantly outperform larger baseline models, highlighting KodCode’s high quality and diversity as a training dataset.

read the caption

Table 2: This table compares the model performances between KodCode-tuned models and strong baseline models across various benchmarks. Average scores are computed by first averaging sub-metrics within each benchmark and then taking the mean across all five benchmarks (HumanEval, MBPP, BCB-Complete, BCB-Instruct, and LiveCodeBench). Bold numbers indicate the best performance for each metric. KodCode-tuned models outperform larger baseline models, demonstrating the dataset’s high quality and diversity.

🔼 This ablation study investigates the impact of data selection on the performance of the Qwen2.5-Coder-32B-Instruct model when fine-tuned on the KodCode-SFT dataset. Three different subsets of the KodCode-SFT dataset were sampled, each containing 10,000 data points. The first subset was a random sample from the full KodCode-SFT dataset (KODCODE-10K). The second contained only the most challenging instances (KODCODE-Hard-10K). The third (NoConvert-10K) excluded data augmented by a style converter. The table compares the performance of the model trained on each subset across four different benchmarks (BigCodeBench-C Full, BigCodeBench-C Hard, BigCodeBench-I Full, BigCodeBench-I Hard, LiveCodeBench Easy, LiveCodeBench Medium, LiveCodeBench Hard). The results show how different data sampling strategies affect the model’s ability to perform well across various difficulty levels and benchmarks.

read the caption

Table 3: Ablation study of data selection when fine-tuning Qwen2.5-Coder-32B-Instruct on KodCode-SFT. Each model is trained on 10K sampled data. Bold numbers indicate best performance for each metric.

🔼 Table 4 presents the hyperparameters used during the supervised fine-tuning process of the Qwen-2.5 model on the KODCODE dataset. It lists the hyperparameter name and its corresponding value, providing details such as the learning rate, number of epochs, batch size, optimizer used (Adamw), learning rate scheduler (cosine), and maximum sequence length. This table is crucial for understanding the specific settings used in the model’s training phase.

read the caption

Table 4: This table shows the hyper-parameters for supervised fine-tuning.