QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension

🔼 QuoTA’s framework consists of four stages: First, a dynamic frame sampler extracts frames from the video based on its duration. Second, a Vision Transformer (ViT) processes these frames to produce visual embeddings. Third, the ‘based LVLM’ uses Chain-of-Thought reasoning to transform the input query into a more specific question. This question is then used to prompt a lightweight ‘scoring LVLM’ to generate frame-wise importance scores, evaluating each frame’s relevance to the query. Finally, a token assigner uses these scores to adjust the visual embeddings, effectively weighting them according to their relevance to the query, producing rescaled frame embeddings.

read the caption

Figure 2: The framework of QuoTA. Initially, a dynamic frame sampler extracts T𝑇Titalic_T frames from the video based on its duration, which are subsequently processed by ViT to generate visual embeddings 𝐄𝐄\bm{\mathrm{E}}bold_E. Then, the based LVLM decouples the input query using Chain-of-Thoughts [36] reasoning into a decoupled clue to generate frame-wise importance scores through scoring LVLM in parallel, thus evaluating the relevance to the query of each frame. Finally, a token assigner rescales the frame embeddings to 𝐄^bold-^𝐄\bm{\mathrm{\hat{E}}}overbold_^ start_ARG bold_E end_ARG based on these importance scores.

🔼 This figure displays a qualitative analysis of QuoTA’s performance on the Video-MME benchmark using the LLaVA-Video-7B model. It showcases two example queries and their corresponding video segments. Frames deemed important by QuoTA (query-oriented keyframes) are highlighted with blue borders. A bar chart visually represents the normalized importance scores assigned by QuoTA to each frame within the video segment, illustrating how the model prioritizes certain frames based on their relevance to the query. This visualization helps understand how QuoTA focuses the model’s attention on the most pertinent parts of the video for accurate response generation.

read the caption

Figure 3: Qualitative result shown in Video-MME [7] benchmark when applying QuoTA with LLaVA-Video-7B [48]. The video frames with a blue border are query-oriented keyframes, and the bar chart shows the normalized scores of QuoTA for each frame.

🔼 This figure shows the detailed prompt engineering strategy used in the paper for object list-based chain of thought reasoning. The prompt guides the large language model (LLM) to first determine if identifying specific objects is necessary to answer the question. If yes, the model is then prompted to list those objects, followed by a filtering step to eliminate any abstract concepts and leave only physical entities. The example prompts help illustrate this three-step process and ensure the LLM focuses on relevant objects in the video frames.

read the caption

Figure 4: CoT-driven decouple prompt for object list.

🔼 This figure demonstrates the Chain-of-Thought (CoT) driven query decoupling prompt specifically designed for video events. The prompt guides the large language model (LLM) to break down a complex query into a simpler, more focused question about the presence of key video events or elements within a frame. This process aids in generating more accurate frame-level importance scores by focusing the LLM on specific, easily identifiable aspects of the video content rather than the entire, potentially ambiguous original query.

read the caption

Figure 5: CoT-driven decouple prompt for video event.

🔼 Figure 6 presents a detailed breakdown of the performance achieved by various frame scoring strategies within the LLaVA-Video-7B model on the Video-MME benchmark. It showcases a comparison of different approaches for assigning importance scores to video frames, including methods based on LVLMs with and without Chain-of-Thought (CoT) reasoning, and even CLIP-based scoring. The graph displays the performance on individual sub-tasks within Video-MME, offering a granular view of how each method performs in aspects such as temporal perception, spatial perception, action recognition, and more. This allows for a direct comparison of the effectiveness of different query-oriented token selection strategies and their relative strengths and weaknesses across a range of video understanding tasks.

read the caption

Figure 6: Sub-task results shown in Video-MME [7] benchmark when applying distinct frame scoring strategy of LLaVA-Video-7B [48].

🔼 Table 2 presents the performance comparison of various 7B-scale Large Video-Language Models (LVLMs) on three video understanding benchmarks: MVBench, VNBench, and NeXT-QA. The results are shown using the original frame rates of the videos. A key finding is that incorporating QuoTA into the LLaVA-Video-7B model leads to a significant average performance improvement (4.8%) across all three benchmarks. The most substantial improvement (10.3%) is observed on the VNBench (Needle-In-A-Haystack) benchmark, demonstrating QuoTA’s effectiveness in focusing on query-relevant keyframes. This places QuoTA’s enhanced LLaVA-Video-7B model ahead of the state-of-the-art model LongVILA.

read the caption

Table 2: The overall performance on MVBench [15], VNBench [49] and NeXT-QA [39] at 7B-scale LVLMs with the setting of the original frame rates. By applying QuoTA to LLaVA-Video-7B [48], we observed an average performance improvement of 4.8% across three benchmarks, especially a 10.3% improvement on the Needle-In-A-Haystack benchmark VNBench [49], which set a new state-of-the-art performing better then LongVILA [40], demonstrating QuoTA assist query-oriented keyframes focusing.

🔼 This table presents an ablation study analyzing the impact of different components of the QuoTA model on two video understanding benchmarks: Video-MME and VNBench. The study compares several configurations of the QuoTA model, varying the frame sampling method (fixed-length sampling of 96 frames versus dynamic sampling of 96-160 frames), the inclusion of LVLM-based frame scoring (Wei.), and the use of Chain-of-Thought driven query decoupling (CoT-Dec.). All configurations maintain a consistent token budget. The results show the performance of each configuration on Video-MME and VNBench, allowing for a detailed analysis of the contribution of each QuoTA component.

read the caption

Table 3: Results on combinations of different components in Video-MME [7] and VNBench [49] when using Long-LLaVA-7B [43] as the based LVLM on QuoTA. Fix-len. and Dy-len. represent fix sampled 96-frame and dynamic sampled 96∼similar-to\sim∼160 frames with the same token budget Ntsubscript𝑁𝑡N_{t}italic_N start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT as the baseline, respectively. Wei. and CoT-Dec. denote the LVLM-based frame scoring and CoT-Driven Query Decouple in Section 3.3 and 3.4, respectively.

🔼 Table 4 shows the performance comparison of different frame sampling strategies and token budgets using the Video-MME and VNBench datasets. The baseline model is LLaVA-Video-7B with a fixed number of frames and tokens. The experiments introduce dynamic frame sampling, adjusting the number of frames based on video length and varying the total token budget. Different combinations of Tbase (base number of frames), α (hyperparameter controlling the maximum number of additional frames), and Nt (total visual token budget) are tested. The table presents results for these varying configurations, showing the impact on performance (measured by accuracy on the benchmarks) and inference time per sample. The gray rows indicate the baseline’s performance with fixed frame sampling.

read the caption

Table 4: Results on different frame-sampling strategies and token budgets in Video-MME [7] and VNBench [49]. Gray rows represent baseline LLaVA-Video-7B [48] with fixed frame sampling while others represent dynamic sampling with varying Tb⁢a⁢s⁢esubscript𝑇𝑏𝑎𝑠𝑒T_{base}italic_T start_POSTSUBSCRIPT italic_b italic_a italic_s italic_e end_POSTSUBSCRIPT and α𝛼\alphaitalic_α in different budget Ntsubscript𝑁𝑡N_{t}italic_N start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT when extending the baseline with QuoTA. #Time denotes the average time during inference per sample.

🔼 This table presents a comparison of different visual token assignment strategies within the QuoTA framework, specifically focusing on their impact on the performance of the LLaVA-Video-7B model across two video understanding benchmarks: Video-MME and VNBench. The strategies evaluated include the baseline (no special token assignment), bilinear interpolation, adaptive pooling, and dynamic token merging. The results highlight the effectiveness of each method in terms of overall performance, broken down further by sub-task within the benchmarks. The ‘None’ row represents the model’s performance without any specialized token assignment, providing a basis for comparison.

read the caption

Table 5: Results on different token assignment strategies when extending QuoTA with LLaVA-Video-7B [48] on Video-MME [7] and VNBench [49]. “None” represents the baseline.

🔼 This table presents a comparative analysis of different frame scoring strategies within the QuoTA framework, specifically when integrated with the LLaVA-Video-7B model. It evaluates the performance on two benchmark datasets: Video-MME and VNBench. The strategies compared include using no frame scoring (baseline), a LVLM-based approach, LVLM-based approaches with Chain-of-Thought (CoT) reasoning for entity or event-centric decomposition, and a CLIP-based CoT approach. The results are shown in terms of overall performance and broken down for different sub-tasks within each benchmark to highlight the strengths and weaknesses of each scoring strategy.

read the caption

Table 6: Results on different frame scoring strategies when extending QuoTA with LLaVA-Video-7B [48] on Video-MME [7] and VNBench [49]. “None” represents the baseline.

🔼 This table presents a quantitative comparison of different frame scoring strategies within the QuoTA framework, specifically when integrated with the LLaVA-Video-7B model on the VNBench benchmark. It details the performance across various sub-tasks, including counting, ordering, and retrieval, allowing for a nuanced evaluation of the effectiveness of distinct scoring approaches. The results showcase the impact of the various methods on achieving the overall goal of accurate and efficient video understanding.

read the caption

Table 7: Results on different frame scoring strategies when extending QuoTA with LLaVA-Video-7B [48] on VNBench [49].