Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for Training-Free Acceleration

🔼 This figure illustrates the three-stage pipeline of the FiCoCo method series for training-free token reduction in Multimodal Large Language Models (MLLMs). It shows how FiCoCo-V and FiCoCo-L, which target the visual and language encoding phases respectively, each apply the filter, correlate, and compress stages. The filter stage identifies redundant tokens. The correlate stage establishes relationships between these and the preserved tokens. Finally, the compress stage integrates the redundant information into the preserved tokens. The figure visually depicts the flow and operations within each stage for both FiCoCo-V and FiCoCo-L, highlighting the differences in their approaches while maintaining a consistent structure across all three stages.

read the caption

Figure 2: An overview of the proposed FiCoCo method series. During different phases of MLLM inference, FiCoCo-V and FiCoCo-L provide distinct solutions across three stages.

🔼 Figure 3 visualizes the effects of token reduction using FiCoCo-V and FiCoCo-L methods. In (a), FiCoCo-V’s token reduction is shown, highlighting how a key visual token (red box) is merged into another (green box). In (b), FiCoCo-L’s token reduction is presented, also demonstrating the merging of a key token (red box) with another token (green box). The process of token merging is tracked visually to show how important information is preserved. This qualitative analysis helps illustrate how the methods maintain relevant information during the reduction process, showing their effectiveness in reducing computational cost without significantly impacting accuracy.

read the caption

Figure 3: Visualizations of token reduction by (a) FiCoCo-V and (b) FiCoCo-L. The red box indicates the traced patch token, while the green box shows where the traced token is merged.

🔼 This figure displays the sensitivity analysis results for three hyperparameters (λ, β, and γ) used in the FiCoCo model. The analysis was performed on two benchmarks: TextVQA and SQA. Each subplot shows how changes in a specific hyperparameter affect the accuracy of the model on both benchmarks. The x-axis represents the value of the hyperparameter, while the y-axis represents the model’s accuracy. The plots provide insights into the optimal ranges and impact of these hyperparameters on the model’s performance, guiding hyperparameter tuning for improved results.

read the caption

Figure 4: Hyperparameter sensitivity analysis of λ𝜆\lambdaitalic_λ, β𝛽\betaitalic_β and γ𝛾\gammaitalic_γ on TextVQA and SQA benchmarks.

🔼 This table presents an ablation study on the FiCoCo-V model, showing the impact of different components on the model’s performance. It breaks down the results by examining the effects of removing the local redundancy calculation, task redundancy calculation, the local penalty strategy, and different fixed values for the hyperparameter K. This allows for a detailed analysis of each component’s contribution to the overall accuracy and efficiency of FiCoCo-V on the SQA and TextVQA benchmarks.

read the caption

Table 2: Ablation results of FiCoCo-V.

🔼 This table presents a comparison of the performance of various methods for accelerating multimodal large language models (MLLMs), specifically focusing on models with 7 billion parameters. It shows the results across ten different benchmark datasets, comparing the performance (accuracy) and computational cost (TFLOPs) of different approaches. The baselines include several existing methods from the literature, while the authors’ methods (FiCoCo-V, FiCoCo-L, and FiCoCo-VL) are also included. Note that because the baselines come from different publications, there might be small variations in reported performance numbers due to differences in experimental setups. The table primarily compares the authors’ methods with other training-free techniques, meaning methods that do not require retraining the model to achieve speed improvements.

read the caption

Table 3: Comparison results on MLLMs with a 7B LLM. For baselines, we reference results reported in other papers, which may exhibit slight discrepancies from the experimental results presented earlier. Our methods are primarily compared with training-free approaches.

🔼 This table presents the ablation study results for the FiCoCo-L model, demonstrating the impact of removing different components on the model’s performance. It analyzes the contribution of each of the three stages (Filter, Correlate, and Compress) and various design choices within those stages on two benchmark datasets, SQA and TextVQA. The results show the effect of removing or modifying elements such as local and task redundancy in the Filter stage, direct and indirect correlation in the Correlate stage, and different compression strategies (e.g., fixed K-values versus adaptive K) in the Compress stage. The table aims to provide a detailed understanding of the individual components’ contribution to the overall model performance.

read the caption

Table 4: Ablation results of FiCoCo-L.

🔼 Table 5 presents a detailed comparison of various methods’ performance on multimodal large language models (MLLMs) using a 13B parameter LLM. It showcases the accuracy achieved by different techniques across ten widely-used benchmark datasets. The table highlights the trade-off between computational efficiency (measured in TeraFLOPs) and accuracy. A key aspect of the table is its focus on comparing training-free methods against existing methods. This is crucial because training-free methods offer a more practical and accessible approach for accelerating the inference of these large models. The results allow for direct comparison between the proposed FiCoCo methods and existing state-of-the-art techniques, demonstrating the effectiveness of the proposed approaches.

read the caption

Table 5: Comparison results on MLLMs with a 13B LLM. For baselines, we reference results reported in other papers. Our methods are primarily compared with training-free approaches.

🔼 This table presents the results of an ablation study evaluating the impact of the hyperparameter ε (epsilon) on the performance of the FiCoCo model. Epsilon controls the threshold for determining which tokens are considered correlated during the compression stage. The table shows how varying epsilon affects the accuracy on two benchmarks: TextVQA and SQA, indicating the optimal setting for epsilon that balances efficiency and accuracy.

read the caption

Table 6: Hyperparameter sensitivity analysis of ε𝜀\varepsilonitalic_ε on TextVQA and SQA benchmarks.

🔼 This table presents the results of an ablation study investigating the impact of the scaling coefficient hyperparameter used in the local penalty strategy within the FiCoCo-V method. The study evaluates performance on two benchmarks: TextVQA and SQA. Different scaling coefficient values are tested to determine their effect on model accuracy. The goal is to identify the optimal balance between preventing spatial-centralized information loss and achieving efficient performance.

read the caption

Table 7: Hyperparameter sensitivity analysis of scaling coefficient in local penalty strategy on TextVQA and SQA benchmarks.

🔼 This table presents a detailed efficiency analysis of various methods for accelerating inference in Multimodal Large Language Models (MLLMs), specifically using the LLaVA-1.5-7B model. It compares the original LLaVA-1.5 model with three variants of the FiCoCo method (FiCoCo-V, FiCoCo-L, FiCoCo-VL) under different quantization levels (FP16, INT8, INT4). The metrics presented include total inference time (TFLOPs), total memory usage (GB), and KV-Cache usage (MB). This allows for a comprehensive comparison of the efficiency gains achieved by FiCoCo in reducing computational cost and memory requirements while maintaining performance.

read the caption

Table 8: Efficiency analysis of methods based on LLaVA-1.5-7B.

🔼 This table presents a comprehensive efficiency analysis of various methods, including the proposed FiCoCo variants, using the LLaVA-1.5-13B model as the base. It compares the performance of these methods across different quantization levels (FP16, INT8, INT4), showing the trade-offs between computational cost (TFLOPs), total memory usage, and KV-cache size. The results highlight the efficiency gains achieved by FiCoCo in reducing computational cost and memory footprint while maintaining comparable accuracy.

read the caption

Table 9: Efficiency analysis of methods based on LLaVA-1.5-13B.

🔼 This table presents a comparison of performance metrics for different models on the Open-LLaVA-NeXT-7B benchmark. The models are categorized based on whether they utilize FlashAttention, a technique for accelerating inference. The key performance indicators (KPIs) presented include FLOPs (floating-point operations), inference time, and accuracy on two specific benchmarks (SQA and MMB). The purpose of the table is to demonstrate that the proposed FiCoCo methods effectively improve efficiency across various scenarios, even when using or not using FlashAttention.

read the caption

Table 10: Comparisons based on Open-LLaVA-NeXT-7B. We categorize the methods based on the availability of FlashAttention and provide FLOPs and time measurements to demonstrate that our methods can effectively accelerate across different scenarios.