CoMP: Continual Multimodal Pre-training for Vision Foundation Models

🔼 Figure 2 illustrates the architecture of the COMP (Continual Multimodal Pre-training) framework. Panel (a) provides a high-level overview of the entire pre-training pipeline, showcasing the integration of a Vision Foundation Model (VFM), a Large Language Model (LLM), and a newly proposed continual multimodal pre-training process. It highlights the two key loss functions: the decoding loss for text generation and the alignment loss for aligning visual and textual representations. Panel (b) focuses specifically on the Continual Rotary Position Embedding (C-ROPE) module, detailing how it handles variable-resolution visual inputs. For clarity, minor components like projection and scaling operations are omitted in the diagram.

read the caption

Figure 2: Architecture of our CoMP. (a) Overview of our pre-training framework; (b) Detail of C-RoPE. For ease of visualization, the projection layers 𝒫⁢r⁢o⁢jq,k,v,o𝒫𝑟𝑜subscript𝑗𝑞𝑘𝑣𝑜\mathcal{P}roj_{q,k,v,o}caligraphic_P italic_r italic_o italic_j start_POSTSUBSCRIPT italic_q , italic_k , italic_v , italic_o end_POSTSUBSCRIPT and scale operators are omitted.

🔼 The Alignment Loss in the figure is calculated using global visual features (Fv) and text features (Ft). These features are projected into the language space using a learned prototype (W), which is the word embedding of Large Language Models (LLMs). The Sinkhorn-Knopp algorithm normalizes the projected features into probability distributions, and then cross-entropy is used to compute the loss, encouraging alignment between visual and textual representations. To avoid information leakage from the image, text features are extracted without the image prefix.

read the caption

Figure 3: Alignment Loss. We illustrate it in the case of one single pair of global vision and text features 𝐅vsubscript𝐅𝑣\mathbf{F}_{v}bold_F start_POSTSUBSCRIPT italic_v end_POSTSUBSCRIPT and 𝐅tsubscript𝐅𝑡\mathbf{F}_{t}bold_F start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT for simplicity. 𝐅vsubscript𝐅𝑣\mathbf{F}_{v}bold_F start_POSTSUBSCRIPT italic_v end_POSTSUBSCRIPT and 𝐅tsubscript𝐅𝑡\mathbf{F}_{t}bold_F start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT are mapped by frozen learned prototype 𝐖𝐖\mathbf{W}bold_W, i.e., the word embedding of LLMs. Then, they are converted into normalized probabilities using the Softmax function and iterative Sinkhorn-Knopp algorithm [12], respectively. Then, cross-entropy is applied as the loss. To prevent information leakage, the text features are extracted without image prefixes.

🔼 This figure analyzes how varying image resolution at inference time affects the performance of the CoMP-MM-1B model on two specific downstream tasks: DocVQA [47] and ChartQA [46]. DocVQA focuses on question answering about document images, while ChartQA involves understanding chart-related queries. Both tasks are sensitive to image resolution. The graph shows that increased resolution leads to improved accuracy. This demonstrates CoMP-MM-1B’s ability to handle high-resolution inputs effectively.

read the caption

Figure 4: Varying the image resolution during inference. We investigate the impact of image resolution on DocVQA [47] and ChartQA [46] by our CoMP-MM-1B.

🔼 This table presents a comprehensive evaluation of the COMP (Continual Multimodal Pre-training) method on various multimodal understanding benchmarks. The experiments were performed using two different Vision Foundation Models (VFMs), CoMP-SigLIP and CoMP-DINOv2, in conjunction with the LLaMA-3.0 8B language model. The vision encoders were frozen, and only the language model was fine-tuned for a single epoch using the LLaVA SFT dataset. The table compares the performance of these models against other state-of-the-art approaches, highlighting the improvements achieved by COMP. The ‘#Patches’ column indicates the number of visual patches processed by the LLM, providing insight into the computational demands of each model. Performance is reported on both training and validation sets, as indicated by the † symbol.

read the caption

Table 2: Evaluation on multimodal understanding benchmarks. We conduct extenseive experiments on CoMP-SigLIP and CoMP-DINOv2 with LLaMA-3.0 8B [19], freezing the vision encoder and directly tuning on LLaVA SFT data [41] for one epoch. #Patches indicates the number of input visual patches for the LLM. ††\dagger† denotes we report the performance on validation sets.

🔼 Table 3 presents the results of image classification experiments conducted on the ImageNet-1K dataset [16]. The experiments utilized a ‘frozen trunk’ approach, meaning that only the final classification layers were trained while the pre-trained weights of the Vision Foundation Model (VFM) were kept unchanged. The performance of several VFMs (including those enhanced by the COMP method) is compared. The classification accuracy is reported for two different input image resolutions: 224x224 pixels and 448x448 pixels. Attentive pooling probing was used as the classification method.

read the caption

Table 3: Evaluation on frozen trunk classification. All experiments are conducted on ImageNet-1K [16] at 224px and 448px by utilizing attentive pooling probing.

🔼 Table 4 presents the results of semantic segmentation experiments conducted on the ADE20K dataset. The experiments involved training the UperNet head while keeping the backbone model frozen. Two different input resolutions (504px and 672px) were used for a comprehensive evaluation. The table compares the performance of various Vision Foundation Models (VFMs) on this task, highlighting the impact of different pre-training methods and model architectures on segmentation accuracy.

read the caption

Table 4: Evaluation on semantic segmentation. All experiments are conducted on ADE20K [82] at 504px and 672px by freezing the backbone and only train the UperNet [74] head.

🔼 This table presents an ablation study on the training recipe used in the paper. It starts with a baseline three-stage training approach, using progressively higher image resolutions (384px, 576px, 768px). Each stage involves different training strategies: In the first stage, only the adapter is trained while the rest of the model is frozen; subsequent stages fine-tune the entire model. The table then systematically varies aspects of this baseline recipe (e.g., when RoPE-2D is introduced, whether the language model is frozen, training data size, etc.), showing the impact of each change on the performance metrics (AI2D, ChartQA, DocVQA). The final, optimal recipe is highlighted in blue. All experiments were conducted using the SigLIP-400M and Qwen2.5-0.5B models.

read the caption

Table 5: Ablation on training recipe. The first row presents the baseline three-stage training recipe, with progressive resolutions of 384px, 576px, and 768px. In the first stage, only the adapter is unfrozen, while in the subsequent stages, the full model is fine-tuned. Each following row modifies strategy from the last non-italicized row, and our final recipe is highlighted in blue. Experiments are run using SigLIP-400M and Qwen2.5-0.5B. ††\dagger†: Unfreezing vision encoder for RoPE training; ΔΔ\Deltaroman_Δ: The average of the difference from previous row.

🔼 This table presents an ablation study on the impact of different positional embedding methods on the performance of a multimodal model. Specifically, it compares the results of using learned positional embeddings at different resolutions (384px and 768px), using Rotary Position Embedding (ROPE)-2D at 768px, and finally using the proposed Continual Rotary Position Embedding (C-ROPE) at 768px. The experiment uses Qwen2-0.5B [76] and SigLIP-400M [80] models, fine-tuned for one epoch on LLaVA-NeXT-SFT [43] data. The evaluation metrics are ChartQA and DocVQA scores.

read the caption

Table 6: Ablation on positional embeddings. We utilize Qwen2-0.5B [76] and SigLIP-400M [80] to be directly fine-tuned on LLaVA-NeXT-SFT [43] data for one epoch.

🔼 This table presents an ablation study on the impact of the Alignment Loss component in the COMP model. Specifically, it shows the performance of the COMP model on three benchmark tasks (ChartQA, DocVQA, and MMMU) under different configurations. Two prominent vision foundation models, Qwen2-0.5B and DINOv2-Large, were used as backbones. The training data consisted of LLaVA-NeXT-SFT for Stage-III. The variations explored were the presence or absence of Alignment Loss, offering a comparison of the model’s performance with and without this crucial component. The results demonstrate the impact of Alignment Loss on model accuracy for multimodal understanding.

read the caption

Table 7: Ablation on the effectiveness of Alignment loss. We utilize Qwen2-0.5B [76] and DINOv2-Large [50], and employ LLaVA-NeXT-SFT [43] as training data of Stage-III.

🔼 This table presents an ablation study on the impact of different vision encoders on the performance of the COMP model. It compares the results of using various vision encoders in terms of their size (Base vs. Large) and pretraining tasks (vision-only vs vision-language). The results are shown for three key metrics (ChartQA, DocVQA, MMMU) demonstrating how different choices for the vision encoder affect the overall performance of the multimodal understanding task.

read the caption

Table 8: Ablation on vision encoders. We ablation the vision encoder of different model sizes and pretraining tasks.

🔼 Table 9 details the configurations used for each of the three training stages (Stage-I, Stage-II, and Stage-III) in the CoMP-MM model training process for both the CoMP-MM-1B and CoMP-MM-7B model variants. It shows the hyperparameters for training the adapter, VFM, and LLM components, specifying details such as batch size, learning rates (for adapter, VFM, and LLM), and the number of epochs for each stage. The table highlights the differences in training configurations between the different stages and between the two model variants.

read the caption

Table 9: Detailed configuration for each training stage of our CoMP-MM-1B and CoMP-MM-7B models.

🔼 Table 10 presents the hyperparameters used for instruction tuning of the CoMP-SigLIP and CoMP-DINOv2 models. It details the optimizer, learning rates (peak and minimum), learning rate schedule, batch size, number of epochs, and warmup epochs. Specific configurations are provided for the adapter and vision foundation model (VFM) components separately. Transformations applied during data augmentation are also listed. This table provides a detailed breakdown of the training settings used to fine-tune the pre-trained vision models with instruction data for improved multimodal understanding.

read the caption

Table 10: Detailed configuration of CoMP-SigLIP and CoMP-DINOv2 in instruction tuning for multimodal understanding.

🔼 Table 11 presents the hyperparameters used for the frozen trunk classification task on ImageNet-1K. It details the optimizer (AdamW), learning rates (peak and minimum), learning rate schedule (cosine decay), batch size, and augmentations used for both CoMP-SigLIP and CoMP-DINOv2 models. The table also specifies image resolutions (224px and 448px) and other training settings such as weight decay, epochs and warmup epochs. This table allows for a clear comparison of the settings used for training these two models on the ImageNet-1K dataset with a focus on evaluating their performance using only the frozen trunk.

read the caption

Table 11: Detailed configuration of CoMP-SigLIP and CoMP-DINOv2 for frozen trunk classification.

🔼 This table details the specific hyperparameters used for semantic segmentation experiments on the ADE20K dataset, using two different vision foundation models: CoMP-SigLIP and CoMP-DINOv2. It breaks down the optimization settings, including the optimizer used (AdamW), learning rate scheduling (poly decay), batch size, number of iterations, warmup iterations, and augmentation techniques (RandomResizedCrop, RandomFlip, PhotoMetricDistortion, and normalization). The table provides a granular view of the training configurations for both models, enabling a comprehensive understanding of the experimental setup for this particular task.

read the caption

Table 12: Detailed configuration of CoMP-SigLIP and CoMP-DINOv2 for semantic segmentation.