2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

🔼 This figure illustrates the process of creating a multimodal textbook from instructional videos. It begins with using LLMs to create a knowledge taxonomy, which is then used to collect and filter 159,000 instructional videos from the internet. A multi-stage pipeline then processes these videos. First, non-instructional videos are removed based on their Automatic Speech Recognition (ASR) transcripts, leaving 75,000 high-quality videos. These videos are then segmented into shorter clips using timestamps from the ASR, removing clips with misaligned visuals and audio. Finally, keyframes are extracted from each clip, and Optical Character Recognition (OCR) is used to extract text and symbols from the keyframes. The result is a multimodal textbook containing 6.5 million keyframes, 259 million ASR tokens, and 500 million OCR tokens, organized in an image-text interleaved format.

read the caption

Figure 2: An illustration of constructing a multimodal textbook from instructional videos. We first instruct LLMs to construct a knowledge taxonomy, then retrieve and filter videos at metadata level, collecting 159K instructional videos. Then a video-to-textbook pipeline is designed for multi-level knowledge extraction. ① We filter out non-instructional videos using ASR transcripts, retaining 75K high-quality videos. ② We use ASR’s timestamp to segment long videos into short clips, discarding those with misaligned visuals and ASR. ③ We detect keyframes from each clip and extract text and symbols by OCR. Our pipeline produces 6.5M keyframes, 259M ASR, and 500M OCR tokens and organizes them into an image-text interleaved textbook.

🔼 This figure displays the effect of shuffling image order within samples on the performance of vision-language models (VLMs). Three different datasets (MMC4-Core-ff, OBELICS, and the authors’ Textbook-6.5M) were used. For each dataset, 20%, 50%, and 100% of the samples were randomly selected, and the image order within those samples was shuffled. The resulting datasets with shuffled images were then used for pretraining VLMs. The y-axis shows the average accuracy across seven benchmark tasks. The x-axis indicates the percentage of samples that underwent image shuffling (0%, 20%, 50%, 100%). The graph visually demonstrates how sensitive each dataset is to changes in image order, offering insights into the influence of image coherence on VLM performance.

read the caption

Figure 3: We randomly select 20%, 50%, and 100% samples from datasets and shuffle the image order within each sample. These datasets with shuffled images are also used for pretraining. The Accuracy denotes the average of seven benchmarks.

🔼 This figure visualizes the structure of the knowledge taxonomy used in the paper. The top part shows a hierarchical tree diagram of six main subjects (Mathematics, Physics, Chemistry, Earth Science, Engineering, and Computer Science). Each subject branches out into several sub-courses, representing more specific areas of study within the subject. Due to space limitations, only the sub-courses with the highest proportions (most videos) are shown for each subject. The bottom part displays bar charts, one for each subject, showing the distribution of knowledge points across the sub-courses within that subject. This gives a visual representation of the breadth and depth of the educational content included in the dataset.

read the caption

Figure 4: Top: We plot six subjects along with their corresponding sub-courses. Due to space constraints, we selectively visualized only the courses with the highest proportions. Bottom: We count the knowledge points distribution belongs to each subject and its course

🔼 This figure shows a visualization of the water cycle. The image depicts the various stages of the water cycle including evaporation from bodies of water, condensation forming clouds, precipitation in the form of rain, surface runoff, infiltration into the ground, and groundwater flow. The illustration highlights the interconnectedness of these processes and how water continuously circulates within the Earth’s system. This example is presented as part of an earth science lesson within the multimodal textbook.

read the caption

Figure 5: A case presented in our textbook illustrates the water cycle within the domain of earth science.

🔼 Figure 6 showcases an example from the textbook illustrating the principles of mechanics in physics. It presents a step-by-step calculation of acceleration using the formula (v-u)/t, where ‘v’ is the final velocity, ‘u’ is the initial velocity, and ’t’ is the time taken. The example problem involves an object initially at rest accelerating to a certain velocity, visually depicted with keyframes and corresponding text explanations of the calculations. Further, it explains the concept of acceleration units and its derivation from velocity and time. The figure demonstrates the use of both visual and textual cues to clarify the physical principle within the context of an instructional video.

read the caption

Figure 6: A case presented in our textbook introducing the principles of mechanics within the domain of physics.

🔼 This figure shows an example from the textbook that uses two objects with different masses (10kg and 100kg) to illustrate the concept of inertia. A constant force of 50N is applied to both objects. The resulting accelerations are calculated using Newton’s second law (F=ma), demonstrating how a larger mass leads to a smaller acceleration. It then expands on the concept by comparing the rotational inertia of a thin hoop and a solid disk with the same mass and radius, prompting reflection on how mass distribution affects rotational inertia. This section visually explains the relationship between mass and inertia, in both linear and rotational motion.

read the caption

Figure 7: A case presented in our textbook introducing the concepts of velocity and acceleration within the context of physics.

🔼 Figure 8 presents a step-by-step solution to a planar geometry problem. The problem involves finding the area of a rectangle inscribed within a semicircle. The solution methodically uses geometric principles, including properties of isosceles right triangles, inscribed angles, and central angles, to determine the dimensions of the rectangle and calculate its area. The figure is highly visual, using diagrams and annotations to guide the reader through each step of the reasoning process. This showcases how the textbook uses a combination of visual and textual explanations to teach mathematical concepts.

read the caption

Figure 8: A case presented in our textbook demonstrates how to solve a question about planar geometry in the domain of mathematics.

🔼 This figure from the section ‘Analysis of Multimodal Textbook’ shows a chemistry lesson explaining the difference between atoms and molecules using examples of helium, hydrogen, and water. It helps to illustrate that atoms are the basic building blocks of matter, while molecules are formed when two or more atoms are chemically bonded. The figure visually depicts atoms (He, Ne) and molecules (H2, O2, CO2, H2O) clarifying pure substances (elements and compounds) using diagrams and text annotations.

read the caption

Figure 9: A case presented in our textbook illustrates the concepts of molecules, atoms, and compounds in the domain of chemistry.

🔼 This figure shows a step-by-step visual explanation of the Depth-First Search (DFS) algorithm, a graph traversal algorithm. The visualization uses a simple graph and highlights the process of visiting nodes, backtracking when reaching dead ends, and ensuring that each node is visited exactly once. The accompanying text explains the choices made during each step of the traversal, illustrating the algorithm’s logic and how it systematically explores the graph. Additionally, pseudocode for the DFS algorithm is provided, further reinforcing the textual and visual explanation.

read the caption

Figure 10: A case presented in our textbook introduces a depth-first search algorithm.

🔼 This table presents the results of fine-tuning the LLaVA-1.5-7B base model using various interleaved datasets, including the multimodal textbook introduced in the paper. The performance of the model is evaluated across seven different benchmarks: four common Visual Question Answering (VQA) tasks and three math-related tasks. The evaluation is performed under few-shot settings (0-shot, 1-shot, 2-shot, and 4-shot), showing the model’s performance with varying amounts of example data. The table allows for a comparison of the multimodal textbook’s effectiveness in pre-training against existing interleaved datasets, highlighting the impact of dataset quality on model performance in different tasks.

read the caption

Table 2: We continued pre-training the base model of LLaVA-1.5-7B using different interleaved datasets. The results are evaluated on 4 common VQA and 3 math-related benchmarks under few-shot settings.

🔼 Table 3 presents the results of experiments using the Idefics-8B model, a Vision-Language Model (VLM) capable of handling multiple images. Unlike the previous Table 2 which used LLaVA-1.5-7B, this table shows results for Idefics-8B under two conditions: continual pre-training (starting from a pre-trained Idefics-8B-base model) and training from scratch (with randomly initialized weights). The evaluation is performed across multiple benchmarks (OKVQA, TextVQA, MathVista, MathVision, MathVerse) and extended to an 8-shot setting, using randomly selected examples for evaluation, a methodology consistent with prior work [16]. The table aims to compare the performance of the Idefics-8B model trained on different datasets, including the multimodal textbook dataset, illustrating the impact of different training methods and dataset characteristics on the model’s capabilities.

read the caption

Table 3: Except for LLaVA, we also pre-train advanced VLMs with multi-image ability (Idefics): continual pretraining from Idefics-8B-base or pre-training from scratch. The evaluations are extended to an 8-shot using randomly selected examples as previous works [16].

🔼 This table presents the results of a ‘Cheat Test’ designed to evaluate the ability of Vision-Language Models (VLMs) to utilize interleaved context. The test replaces one or two few-shot examples within the prompt with the test sample itself. This assesses if the VLMs recognize the identical image, question, and answer combination and answer efficiently without further processing. The results compare the performance of various models (MMC4-cf, OBELICS, and the authors’ model) across multiple benchmarks (OKVQA, TextVQA, MathVista, MathVision, MathVerse) under 1-shot and 2-shot conditions. The goal is to understand how well the models leverage their interleaved context for accurate and efficient responses.

read the caption

Table 4: We design “Cheat Test” to observe whether VLMs can attend to their interleaved context. We replace a few-shot example with the test sample itself and observe whether VLM notice this identical <<>> within their prompt. Itsubscript𝐼𝑡I_{t}italic_I start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT, qtsubscript𝑞𝑡q_{t}italic_q start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT, atsubscript𝑎𝑡a_{t}italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT denote the test case, Iesubscript𝐼𝑒I_{e}italic_I start_POSTSUBSCRIPT italic_e end_POSTSUBSCRIPT, qesubscript𝑞𝑒q_{e}italic_q start_POSTSUBSCRIPT italic_e end_POSTSUBSCRIPT, aesubscript𝑎𝑒a_{e}italic_a start_POSTSUBSCRIPT italic_e end_POSTSUBSCRIPT denote a random selected example.

🔼 This table presents the zero-shot performance results of three different vision-language models (VLMs) after instruction fine-tuning. The models were fine-tuned using 665K data from the LLaVA-1.5 dataset. The table compares the performance on two visual question answering (VQA) benchmarks (OKVQA and TextVQA) and one math-related benchmark (MathVista). The results show the improvement in performance after instruction fine-tuning for each model on the specified benchmarks. This demonstrates the impact of fine-tuning and allows for comparison of different models’ performance on the same benchmarks.

read the caption

Table 5: We also evaluated the zero-shot result after instruction fine-tuning using the 665K data from LLaVA-1.5.

🔼 This table presents an ablation study on the process of creating the multimodal textbook dataset. It shows the impact of different components of the pipeline on the final dataset performance. Specifically, it analyzes the effects of refining the automatically generated speech-to-text (ASR) transcripts, the incorporation of optical character recognition (OCR) to extract text from images, and the choice of algorithm used for selecting keyframes from video clips. The results demonstrate the contribution of each step to overall performance metrics.

read the caption

Table 6: We perform an ablation study on video-to-textbook pipeline, including the impact of ASR refinement, the necessity of incorporating OCR, and the algorithms for extracting keyframes.

🔼 Table 7 provides a detailed statistical overview of the multimodal textbook dataset used in the paper. It breaks down the dataset’s composition across six subjects (Mathematics, Physics, Chemistry, Earth Science, Engineering, and Computer Science). For each subject, the table shows the number of videos, the total duration of those videos in hours, the number of topics covered, the number of video clips extracted, the number of keyframes extracted, and the counts of Automatic Speech Recognition (ASR) tokens and Optical Character Recognition (OCR) tokens. Finally, it indicates the total number of samples generated for model training. This information offers insights into the scale and characteristics of the dataset, highlighting its richness and diversity in terms of video content, extracted textual data, and the resulting training samples.

read the caption

Table 7: The statistics of our multimodal textbook. Topic denotes the knowledge points covered by each category of videos, which are sourced from our knowledge taxonomy.