Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

🔼 InternVL 2.5 uses a ‘ViT-MLP-LLM’ architecture. InternViT (a Vision Transformer) processes images, reducing the initial 1024 visual tokens to 256 using pixel unshuffle. These tokens are then projected via an MLP (Multilayer Perceptron) into an LLM (Large Language Model) for multimodal understanding. Unlike earlier versions, InternVL 2.5 supports multi-image and video inputs.

read the caption

Figure 2: Overall architecture. InternVL 2.5 retains the same model architecture as InternVL 1.5 [35] and InternVL 2.0, i.e. the widely-used “ViT-MLP-LLM” paradigm, which combines a pre-trained InternViT-300M or InternViT-6B with LLMs [19, 229] of various sizes via an MLP projector. Consistent with previous versions, we apply a pixel unshuffle operation to reduce the 1024 visual tokens produced by each 448×\times×448 image tile to 256 tokens. Moreover, compared to InternVL 1.5, InternVL 2.0 and 2.5 introduced additional data types, incorporating multi-image and video data alongside the existing single-image and text-only data.

🔼 Figure 3 illustrates how the InternVL model handles different data types: (a) Single-image inputs are divided into tiles, with the maximum number of tiles used to ensure the highest resolution. (b) Multi-image inputs distribute tiles proportionally among the images in a sample. (c) Video processing simplifies to resizing individual frames to 448x448 pixels.

read the caption

Figure 3: Illustration of the data formats for various data types. (a) For single-image datasets, the maximum number of tiles nmaxsubscript𝑛maxn_{\text{max}}italic_n start_POSTSUBSCRIPT max end_POSTSUBSCRIPT is allocated to a single image, ensuring maximum resolution for the input. (b) For multi-image datasets, the total number of tiles nmaxsubscript𝑛maxn_{\text{max}}italic_n start_POSTSUBSCRIPT max end_POSTSUBSCRIPT is distributed proportionally across all images within the sample. (c) For video datasets, the method simplifies the approach by setting nmax=1subscript𝑛max1n_{\text{max}}=1italic_n start_POSTSUBSCRIPT max end_POSTSUBSCRIPT = 1, resizing individual frames to a fixed resolution of 448×\times×448.

🔼 This figure illustrates the training process of the InternVL 2.5 model, highlighting its two key strategies: single model training and progressive scaling. The single model training pipeline involves three stages: a warmup stage focusing on the MLP projector, an optional incremental learning stage for the vision transformer (ViT), and a final instruction tuning stage for the full model. This multi-stage approach improves vision-language alignment, enhances training stability, and prepares the model for integration with larger language models. The progressive scaling strategy leverages the pre-trained ViT module from earlier stages, allowing for easy integration with larger language models, leading to scalable model alignment and reduced computational costs. This figure helps explain the efficient and scalable training methods used for InternVL 2.5.

read the caption

Figure 4: Illustration of the training pipeline and progressive scaling strategy. (a) Single model training pipeline: The training process is divided into three stages—Stage 1 (MLP warmup), optional Stage 1.5 (ViT incremental learning), and Stage 2 (full model instruction tuning). The multi-stage design progressively enhances vision-language alignment, stabilizes training, and prepares modules for integration with larger LLMs. (b) Progressive scaling strategy: The ViT module trained with a smaller LLM in earlier stages can be easily integrated with larger LLMs, enabling scalable model alignment with affordable resource overhead.

🔼 Figure 5 details the configuration of datasets used to train InternVL 2.0 and 2.5. Data augmentation techniques (like JPEG compression) are selectively applied; they are used for image data but not video or text data. The maximum tile number (nmax) parameter determines the resolution of the input; higher nmax values are used for higher-resolution inputs, such as those found in multi-image datasets. Conversely, lower nmax values are used for video data, which often has many frames to process. The repeat factor (r) controls the sampling frequency of each dataset, balancing dataset representation and ensuring robust and balanced model training.

read the caption

Figure 5: Dataset configuration. In InternVL 2.0 and 2.5, data augmentation is applied selectively, enabled for image datasets and disabled for videos and text. The maximum tile number (nmaxsubscript𝑛n_{\max}italic_n start_POSTSUBSCRIPT roman_max end_POSTSUBSCRIPT) controls the resolution of inputs, with higher values for multi-image datasets and lower values for videos. The repeat factor (r𝑟ritalic_r) balances dataset sampling by adjusting the frequency of each dataset, ensuring robust and balanced training.

🔼 Figure 6 showcases examples of problematic data points (abnormal samples) frequently found within open-source datasets used to train large language models. These issues affect various data types including single images, multiple images, videos, and text-only datasets. A major problem highlighted is the prevalence of repetitive outputs within the data. The authors argue that these repetitive patterns are highly detrimental to the performance of models, particularly during test-time scaling, often causing them to produce repetitive or cyclical responses, especially in long-form outputs and when using Chain-of-Thought (CoT) reasoning.

read the caption

Figure 6: Visualization of abnormal samples in open-source datasets. Abnormal samples are prevalent across various data types, including single-image, multi-image, video, and pure text datasets, with “repetitive outputs” being a prominent issue. We identify this as one of the most detrimental problems for test-time scaling, often leading models into loops in long-form outputs and CoT reasoning tasks.

🔼 This figure shows the growth of the dataset used for fine-tuning the InternVL model from version 1.5 to version 2.5. It displays the increase in both the number of samples and the number of tokens across various data types: single images, multiple images, videos, and text. The growth indicates an increase in the scale and diversity of the training data, which ultimately enhances the model’s ability to understand and process multiple data modalities.

read the caption

Figure 7: Statistics of the fine-tuning data mixture. The dataset shows consistent growth from InternVL 1.5 to 2.5 in terms of (a) the number of samples and (b) the number of tokens across multiple dataset types, including single-image, multi-image, video, and text. These statistics reflect iterative improvements in data scale and diversity, which enhance the model’s multimodal understanding capabilities.

🔼 This figure illustrates the data filtering pipeline used to improve the quality of the training data. For text data, a three-stage process is used: LLM-based quality scoring to filter out low-quality samples based on domain-specific scores; repetition detection to remove samples with repetitive patterns; and heuristic rule-based filtering to identify and remove anomalous samples using predefined rules. For multimodal data, the LLM-based quality scoring stage is skipped, and only repetition detection and heuristic rule-based filtering are applied to ensure data integrity and remove repetitive patterns.

read the caption

Figure 8: Dataset filtering pipeline. For text data, we use three methods: (a) LLM-based quality scoring to assign domain-specific quality scores and filter low-quality samples; (b) Repetition detection to identify and remove data with repetitive patterns; and (c) heuristic rule-based filtering to detect anomalies using predefined rules. For multimodal data, only (b) repetition detection and (c) heuristic rule-based filtering are applied to mitigate repetitive patterns and ensure dataset integrity.

🔼 This figure showcases the Chain of Thought (CoT) prompts utilized in the InternVL 2.5 model testing. The prompts are designed to guide the model’s reasoning process step-by-step, enhancing its ability to solve complex problems. The figure likely shows examples of both multiple-choice and open-ended question prompts, illustrating how the CoT approach structures the input to elicit a more detailed and logical reasoning process from the model, ultimately improving its performance, particularly on the MMMU benchmark.

read the caption

Figure 9: CoT prompts used in our model testing. By leveraging these prompts for CoT reasoning, we can scale up testing time, significantly enhancing the performance of InternVL 2.5 models on MMMU [289].

🔼 This figure shows the performance of various models on the LongVideoBench benchmark as the number of input video frames increases. It demonstrates how the accuracy of different models, including InternVL 2.5 models and several other state-of-the-art models, changes with varying frame counts (16, 32, 48, 64, and 128 frames). This visualization helps to understand the impact of temporal information on video understanding tasks, particularly for assessing the scalability of models when processing long videos.

read the caption

Figure 10: Performance on LongVideoBench with varying input video frames.

🔼 This table presents the pre-trained models used across different InternVL versions (1.5, 2.0, and 2.5). InternVL 2.5 models show improvements in both vision encoders and language models compared to previous versions. The table lists each model’s name, number of parameters (#Param), vision encoder used, language model employed, and the corresponding OpenCompass average score. Scores for InternVL 1.5 and InternVL 2.0 are from the OpenCompass leaderboard, while InternVL 2.5 scores are from local testing.

read the caption

Table 2: Pre-trained models used in the InternVL series. In the InternVL 2.5 series, we upgraded both the vision encoder and the language model, resulting in improved performance. The OpenCompass scores for InternVL 1.5 and InternVL 2.0 were collected from the OpenCompass leaderboard, while the scores for InternVL 2.5 series were obtained through our local testing.

🔼 Table 3 details the training configurations and hyperparameters used for various InternVL 2.5 model sizes (1B, 2B, 4B, 8B, 26B, 38B, and 78B parameters). It shows the dataset used (a mixture of datasets), the trainable parameters at each stage (MLP warmup, optional ViT incremental learning, and full model instruction tuning), and hyperparameters such as batch size, learning rate, context length, dropout rate, and weight decay. The number of training tokens is also given for each model. Importantly, it highlights the efficiency of InternVL 2.5 training, showcasing that InternVL2.5-78B required only ~120B tokens for training, significantly less than the 1.4 trillion tokens used in Qwen2-VL [246].

read the caption

Table 3: Training configurations and hyperparameters for InternVL 2.5. This table presents the training setups for various scales of InternVL 2.5 models. The configurations are carefully optimized to ensure efficient scaling and performance across different parameter sizes and training stages. Notably, Qwen2-VL [246] processed a cumulative total of 1.4T tokens, while our InternVL2.5-78B is trained on just ∼similar-to\sim∼120B tokens.

🔼 Table 4 details the composition of the InternVL 2.5 model’s pre-training data. It highlights the exclusive use of conversation-format instruction data and specifies that during this stage, only the Multi-Layer Perceptron (MLP) parameters, or both MLP and Vision Transformer (ViT) parameters, are trainable. This training approach allows for the inclusion of both low- and high-quality data in the pre-training phase.

read the caption

Table 4: Summary of the pre-training data mixture of InternVL 2.5. Notably, we exclusively use conversaiton-format instruction data, and at this stage, only the MLP or both MLP and ViT parameters are trainable, allowing the incorporation of both low- and high-quality data.

🔼 Table 5 details the composition of the fine-tuning dataset used to train InternVL 2.5. The dataset is a multi-lingual blend, primarily English and Chinese, but also incorporates smaller amounts of data in Korean, Japanese, Italian, Russian, German, French, Thai, Arabic, and Vietnamese. The dataset is a mixture of open-source datasets and data that the authors themselves created.

read the caption

Table 5: Summary of the fine-tuning data mixture of InternVL 2.5. We expanded our fine-tuning data mixture through extensive collection of open-source datasets and self-synthesized data. This mixture is predominantly in English (en) and Chinese (zh), with smaller portions in other languages, including Korean (ko), Japanese (ja), Italian (it), Russian (ru), German (de), French (fr), Thai (th), Arabic (ar), and Vietnamese (vi).

🔼 Table 6 presents a comparative analysis of various Multimodal Large Language Models (MLLMs) across several benchmarks that assess multimodal reasoning and mathematical capabilities. The benchmarks include MMMU and MMMU-Pro, which evaluate multidisciplinary reasoning skills across various academic fields, and MathVista, MATH-Vision, MathVerse, and OlympiadBench, which focus specifically on mathematical problem-solving skills. The table highlights the models’ performance on each benchmark, showing their relative strengths and weaknesses in these critical areas of multimodal intelligence. Some scores are taken from other publications and the OpenCompass leaderboard.

read the caption

Table 6: Comparison of multimodal reasoning and mathematical performance. MMMU [289] and MMMU-Pro [290] are multidisciplinary reasoning benchmarks, while MathVista [163], MATH-Vision [245], MathVerse [299], and OlympiadBench [80] are mathematics benchmarks. Part of results are collected from [54, 8, 290, 245, 299, 80] and the OpenCompass leaderboard [46].

🔼 Table 7 presents a comprehensive evaluation of InternVL 2.5’s performance on various OCR, chart, and document understanding tasks. It compares InternVL 2.5 to several other leading open-source and closed-source models across nine diverse benchmarks. These benchmarks assess different aspects of visual and textual understanding, including text recognition, visual question answering, document question answering, and reasoning with chart data. The results highlight InternVL 2.5’s competitive performance and showcase improvements over previous versions of InternVL.

read the caption

Table 7: Comparison of OCR, chart, and document understanding performance. We evaluate OCR-related capabilities across 9 benchmarks, including AI2D [109], ChartQA [181], TextVQA [212], DocVQA [184], InfoVQA [183], OCRBench [158], SEED-2-Plus [125], CharXiv [257], and VCR [302]. Part of results are collected from [64, 54, 8, 257, 302] and the OpenCompass leaderboard [46].

🔼 Table 8 presents a comparative analysis of various multimodal large language models (MLLMs) across a range of benchmarks focusing on multi-image and real-world understanding. The multi-image benchmarks assess the models’ ability to process and reason with multiple images simultaneously. These include BLINK, Mantis-Eval, MMIU, MuirBench, MMT-Bench, and MIRB, each evaluating different aspects of multi-image comprehension. Real-world benchmarks evaluate model performance on more practical, complex scenarios using real-world data; these include RealWorldQA, MME-RealWorld, WildVision, and R-Bench, assessing capabilities like spatial understanding and robustness to real-world image distortions. A subset of the results are drawn from existing literature while others are obtained through local testing.

read the caption

Table 8: Comparison of multi-image and real-world understanding performance. Multi-image benchmarks include BLINK [70], Mantis-Eval [100], MMIU [186], MuirBench [241], MMT-Bench [277], and MIRB [308]. Real-world benchmarks encompass RealWorldQA [47], MME-RealWorld [306], WildVision [171], and R-Bench [126]. Part of the results are sourced from the benchmark papers and the OpenCompass leaderboard [46].

🔼 Table 9 presents a comprehensive evaluation of InternVL 2.5’s performance on various multimodal understanding and hallucination benchmarks. The multimodal understanding benchmarks assess the model’s ability across diverse tasks requiring integrated visual and language processing capabilities. These include MME, MMBench, MMVet, and MMStar. The hallucination benchmarks focus on evaluating the model’s tendency to generate inaccurate or nonsensical outputs, and encompass HallusionBench, MMHal, CRPE, and POPE. The results shown are sourced from both the individual benchmark papers and the OpenCompass leaderboard, allowing comparison to other state-of-the-art models.

read the caption

Table 9: Comparison of comprehensive multimodal understanding and hallucination performance. Comprehensive multimodal benchmarks include MME [68], MMBench series [156], MMVet series [283, 284], and MMStar [28]. Hallucination benchmarks encompass HallusionBench [77], MMHal [223], CRPE [250], and POPE [139]. Part of the results are sourced from the benchmark papers and the OpenCompass leaderboard [46].

🔼 Table 10 presents a detailed comparison of InternVL 2.5’s visual grounding performance against other state-of-the-art models on three benchmark datasets: RefCOCO, RefCOCO+, and RefCOCOg. These datasets evaluate a model’s ability to locate objects within an image based on textual descriptions, with varying levels of complexity and detail in those descriptions. The table highlights InternVL 2.5’s performance across different model sizes (8B and 78B parameters), showcasing its improvement over previous versions and competitiveness with leading models. The results demonstrate the impact of model scaling and architecture on visual grounding capabilities.

read the caption

Table 10: Comparison of visual grounding performance. We evaluate InternVL’s visual grounding capability on RefCOCO, RefCOCO+, and RefCOCOg datasets [108, 177]. Parts of the results are collected from [246].

🔼 Table 11 presents a comprehensive evaluation of the model’s multilingual capabilities across three distinct benchmarks: MMMB, Multilingual MMBench, and MTVQA. Each benchmark assesses performance across six languages: English, Chinese, Portuguese, Arabic, Turkish, and Russian. The table allows for a detailed comparison of the model’s strengths and weaknesses in handling various language-specific aspects within multimodal tasks.

read the caption

Table 11: Comparison of multimodal multilingual performance. We evaluate multilingual capabilities across 3 benchmarks, including MMMB [218], Multilingual MMBench [218] and MTVQA [227]. The languages evaluated are English (en), Chinese (zh), Portuguese (pt), Arabic (ar), Turkish (tr), and Russian (ru).

🔼 Table 12 presents a comprehensive comparison of InternVL’s video understanding capabilities against other state-of-the-art models. It evaluates performance across six different benchmarks, each assessing various aspects of video comprehension. For four benchmarks (Video-MME, MMBench-Video, MLVU, and LongVideoBench), performance is tested using four frame settings (16, 32, 48, and 64), with the maximum result reported. The remaining two benchmarks (MVBench and CG-Bench) use fixed frame settings of 16 and 32 respectively. The table provides a detailed performance analysis across a range of models and benchmarks, allowing for a nuanced understanding of InternVL’s strengths and limitations in video understanding tasks.

read the caption

Table 12: Comparison of video understanding performance. We evaluate InternVL’s video understanding capabilities across 6 benchmarks. For Video-MME [69], MMBench-Video [65], MLVU [315], and LongVideoBench [262], we test with four different settings: 16, 32, 48, and 64 frames, and report the maximum results. For MVBench [131], we conduct testing using 16 frames. For CG-Bench [7], we use 32 frames.

🔼 This table presents a comprehensive comparison of InternVL 2.5 and its predecessor, InternVL 2.0, along with several other LLMs and MLLMs, across a variety of language-centric benchmarks. The benchmarks cover a wide range of tasks, including commonsense reasoning, mathematical problem solving, coding challenges, and general knowledge tests. The results highlight the improvements in pure language capabilities achieved in InternVL 2.5 by employing a larger, higher-quality dataset during training and enhanced filtering to remove low-quality data. The table offers a detailed performance comparison, enabling a clear understanding of InternVL 2.5’s strengths and weaknesses in various aspects of language understanding.

read the caption

Table 13: Comparison of language capabilities across multiple benchmarks. These results were obtained using the OpenCompass toolkit for testing. Training InternVL 2.0 models led to a decline in pure language capabilities. InternVL 2.5 addresses this by collecting more high-quality open-source data and filtering out low-quality data, achieving better preservation of pure language performance.

🔼 Table 14 presents a comprehensive analysis of InternViT’s image classification performance across its various versions (InternViT-6B-224px, InternViT-6B-448px-V1.0, InternViT-6B-448px-V1.2, InternViT-6B-448px-V1.5, InternViT-6B-448px-V2.5). The models were trained using the ImageNet-1K dataset [56] and evaluated not only on the IN-1K validation set but also on several challenging ImageNet variants known for their difficulty: IN-ReaL [16], IN-V2 [199], IN-A [87], IN-R [84], and IN-Sketch [242]. The table reports the average classification accuracy for each InternViT version using two different probing methods: linear probing and attention pooling probing. The difference (Δ) between the accuracies of the attention pooling and linear probing methods highlights the model’s ability to learn increasingly complex, non-linear semantic representations as its architecture evolves.

read the caption

Table 14: Image classification performance across different versions of InternViT. We use IN-1K [56] for training and evaluate on the IN-1K validation set as well as multiple ImageNet variants, including IN-ReaL [16], IN-V2 [199], IN-A [87], IN-R [84], and IN-Sketch [242]. Results are reported for both linear probing and attention pooling probing methods, with average accuracy for each method. ΔΔ\Deltaroman_Δ represents the performance gap between attention pooling probing and linear probing, where a larger ΔΔ\Deltaroman_Δ suggests a shift from learning simple linear features to capturing more complex, nonlinear semantic representations.

🔼 Table 15 presents a detailed comparison of semantic segmentation performance across various versions of the InternViT model. InternViT models were evaluated using two datasets, ADE20K and COCO-Stuff-164K. Three different training configurations were used: linear probing (only training a linear classifier on top of a frozen InternViT), head tuning (training only the UperNet head while keeping InternViT frozen), and full tuning (training both InternViT and the UperNet head). The table shows the mean Intersection over Union (mIoU) scores for each configuration on both datasets. The Δ₁ column represents the difference in mIoU between the head tuning and linear probing methods, while Δ₂ represents the difference in mIoU between full tuning and linear probing. Larger Δ values suggest the model’s representations are shifting from simple linear features to more complex, non-linear features, indicating increased model sophistication and ability to capture complex semantic information.

read the caption

Table 15: Semantic segmentation performance across different versions of InternViT. The models are evaluated on ADE20K [313] and COCO-Stuff-164K [18] using three configurations: linear probing, head tuning, and full tuning. The table shows the mIoU scores for each configuration and their averages. Δ1subscriptΔ1\Delta_{1}roman_Δ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT represents the gap between head tuning and linear probing, while Δ2subscriptΔ2\Delta_{2}roman_Δ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT shows the gap between full tuning and linear probing. A larger ΔΔ\Deltaroman_Δ value indicates a shift from simple linear features to more complex, nonlinear representations.