Craw4LLM: Efficient Web Crawling for LLM Pretraining

🔼 The figure shows the correlation between pretraining influence scores and indegrees on randomly sampled ClueWeb22-B documents. Specifically, subplot (a) displays a weak negative correlation between pretraining influence scores and indegrees, indicating that documents with many incoming links are not necessarily the most influential for pretraining. The Spearman correlation coefficient is reported as -0.11, suggesting that a higher indegree does not strongly imply higher pretraining influence.

read the caption

(a) Pretraining (-0.11).

🔼 The figure shows the correlation between the pretraining influence scores and PageRank scores. The scatter plot visually represents the relationship between these two metrics, calculated on a randomly sampled subset of documents from the ClueWeb22-B dataset. The Spearman correlation coefficient (0.88) indicates a strong positive correlation, suggesting that documents with high PageRank scores tend to also have high pretraining influence scores.

read the caption

(b) PageRank (0.88).

🔼 This figure displays scatter plots illustrating the correlation between pretraining influence scores and other metrics on a subset of documents from the ClueWeb22-B dataset. The pretraining influence scores are derived from the DCLM fastText model. The plots visually compare the relationships between these scores and PageRank, as well as the relationship between PageRank and the number of inlinks (indegree). Spearman correlation coefficients are provided to quantify the strength of each correlation.

read the caption

Figure 2: Correlations between pretraining influence scores from DCLM fastText (Li et al., 2024) and PageRank to indegrees, on randomly sampled ClueWeb22-B documents (Overwijk et al., 2022). Spearman correlation coefficients are reported in parentheses.

🔼 This figure shows the efficiency of different crawling methods in achieving comparable performance to the oracle method. The x-axis represents the pool size (relative to the oracle), indicating how much more data the crawlers processed than the oracle. The y-axis shows the performance of the LLMs trained on the data collected by each crawler. CRAW4LLM achieves nearly the same performance as the oracle by crawling significantly less data, while the other methods, including extended indegree and random crawling, require much larger amounts of data to reach comparable performance.

read the caption

(a) Extended crawling.

🔼 Figure 3(b) shows the number of documents visited and crawled by the CRAW4LLM crawler and baseline crawlers (Indegree and Random). It compares the number of documents crawled (P) to achieve a certain performance level against the number of documents visited (V) during the crawling process. This visualization highlights the efficiency of CRAW4LLM in achieving comparable performance with significantly fewer crawled and visited documents than traditional methods.

read the caption

(b) Visited documents.

🔼 Figure 3 delves into the efficiency of different web crawling methods. Subfigure (a) compares the performance of Large Language Models (LLMs) trained using data collected by Craw4LLM and traditional crawlers (extended baselines). The traditional crawlers are extended to crawl more data in an attempt to match Craw4LLM’s results. Subfigure (b) shows a quantitative comparison of the number of web pages crawled and visited by Craw4LLM versus the estimated number a traditional, indegree-based crawler would need to crawl to achieve similar LLM performance. This demonstrates Craw4LLM’s effectiveness in selecting high-quality data with far fewer crawls.

read the caption

Figure 3: Efficiency of crawlers. (a) shows the performance of LLMs trained on selected data crawled by Craw4LLM and extended baseline crawlers. (b) presents the number of crawled (𝒫𝒫\mathcal{P}caligraphic_P) and visited (𝒱𝒱\mathcal{V}caligraphic_V) documents for Craw4LLM, along with the estimated number of crawled documents required for indegree-based crawler to match Craw4LLM’s performance.

🔼 Figure 4 illustrates the effectiveness of CRAW4LLM in identifying high-quality documents for LLM pretraining. The left panel shows the precision (the percentage of truly valuable documents among those identified by each method) while the right panel shows the recall (the percentage of valuable documents found by each method). CRAW4LLM demonstrates high precision, quickly identifying nearly all the best documents. In contrast, the indegree-based and random crawling methods perform significantly worse. The upper bound line represents a hypothetical ideal scenario where only the best documents are collected.

read the caption

Figure 4: Precision (left) and recall (right) of the oracle documents among the documents crawled by Craw4LLM, indegree, and random crawler. The upper bound represents always crawling the oracle documents.

🔼 This table lists the hyperparameters used for training the language model. It details the architecture of the model, including the number of layers, attention heads, the width of the model’s hidden layers (dmodel), and the width of each attention head (dhead). These parameters significantly influence the model’s capacity and performance.

read the caption

Table 2: Model and training hyper-parameters. nlayerssubscript𝑛layersn_{\text{layers}}italic_n start_POSTSUBSCRIPT layers end_POSTSUBSCRIPT, nlayerssubscript𝑛layersn_{\text{layers}}italic_n start_POSTSUBSCRIPT layers end_POSTSUBSCRIPT, dmodelsubscript𝑑modeld_{\text{model}}italic_d start_POSTSUBSCRIPT model end_POSTSUBSCRIPT, and dheadsubscript𝑑headd_{\text{head}}italic_d start_POSTSUBSCRIPT head end_POSTSUBSCRIPT denote the number of layers, attention heads, width, and width per attention head, respectively.

🔼 This table presents the performance of different web crawling methods on commonsense reasoning tasks. It compares the performance of a baseline method (crawling a large amount of data and then selecting the top-performing subset) using random selection and indegree-based selection with the proposed CRAW4LLM method. The results are shown for different amounts of crawled data (1x, 2x, 4x) relative to the amount used in the CRAW4LLM method (1x). An oracle selection result (selecting the best 10% of data from the entire dataset) serves as an upper bound performance measure. The commonsense reasoning tasks evaluated are CommonsenseQA, COPA, OpenBookQA, and PIQA.

read the caption

Table 3: Results for commonsense reasoning tasks.

🔼 This table presents the performance of different Language Models (LMs) on various language understanding tasks. The models were pre-trained using data obtained from different crawling methods: a baseline crawler using indegree as a priority metric, a random crawler, and the proposed CRAW4LLM method. The table shows the performance (accuracy) of each model across several benchmarks, including BIG-Bench, HellaSwag, LAMBADA, Winograd, and Winogrande. The ‘Pool Size’ column indicates how many times more data was crawled compared to the target training dataset size of 1x. The ‘Oracle Selection (Upper Bound)’ row shows the best possible result achievable by selecting the top 10% of data, based on the DCLM fastText classifier.

read the caption

Table 4: Results for language understanding tasks.

🔼 This table presents the results of the reading comprehension tasks, evaluated using three metrics: BoolQ, CoQA, and SQUAD. It compares the performance of Language Models (LLMs) trained on data obtained through different crawling methods and data selection strategies. The methods include a baseline using random selection, indegree-based crawling, and the proposed CRAW4LLM method, each with varying amounts of crawled data (1x, 2x, and 4x the size of the target dataset). An oracle selection represents the upper bound performance. The table shows the performance (accuracy) achieved by each method for each reading comprehension metric.

read the caption

Table 5: Results for reading comprehension tasks.

🔼 This table presents the performance of different web crawling methods on symbolic problem-solving tasks. It compares the results of using an oracle selection (the upper bound of performance), random crawling, indegree-based crawling (representing traditional graph-connectivity methods), and the proposed CRAW4LLM method. The table shows the performance across several different datasets and tasks, indicating how each crawling strategy impacts the downstream performance of LLMs trained on the data collected.

read the caption

Table 6: Results for symbolic problem solving tasks.

🔼 This table presents the results of evaluating Large Language Models (LLMs) on world knowledge tasks. Different crawling methods (random, indegree-based, and CRAW4LLM) were used to gather data for LLM pretraining, with varying amounts of data collected (1x, 2x, and 4x the target dataset size for crawl-then-select methods). The ‘Oracle Selection’ row represents an upper bound, using the top-performing 10% of data. The table shows performance metrics for each method across several world knowledge benchmarks: ARC Easy, ARC Challenge, BIG-Bench QA, Wikidata, Jeopardy, and MMLU. This allows comparison of the efficiency of different crawling strategies in obtaining high-quality pretraining data that results in strong LLM performance on world knowledge tasks.

read the caption

Table 7: Results for world knowledge tasks.