LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer

🔼 LLaVA-UHD v2 processes image slices and an overview image with a CLIP-ViT. The resulting visual features are passed to a hierarchical window transformer (Hiwin Transformer) that builds an inverse feature pyramid and compresses it into visual tokens. These tokens are then fed to an LLM, facilitating vision-language alignment.

read the caption

Figure 2: The overall architecture of proposed LLaVA-UHD v2, consisting of a ViT, our hierarchical window transformer (Hiwin transformer), and an LLM. Hiwin transformers process sliced patches and the overview image by capturing inner multi-level representations and compressing them into spatially consistent tokens for a better vision-language alignment.

🔼 The Joint Bilateral Upsampling (JBU) module guides feature upsampling by leveraging the image pyramid, effectively integrating high-frequency details into upsampled feature maps. This addresses the issue of standard upsampling methods failing to introduce fine-grained details, which are crucial for tasks demanding high-resolution visual information. The JBU module learns convolutional layers on progressively downsampled image versions. These learned kernels capture local high-frequency textures that guide the upsampling of the initial feature map. Each pixel’s value in the upsampled map is a weighted average of values from the original feature map. These weights are determined by both the spatial distance between pixels and their similarity in terms of local textural patterns captured by the image pyramid. This process results in upsampled feature maps enriched with detailed texture information, making them more suitable for high-resolution tasks.

read the caption

Figure 3: Flowchart of the Joint Bilateral Upsampling (JBU) module, which leverages the image pyramid to guide feature up-sampling, integrating high-frequency information into the up-sampled feature maps.

🔼 The figure illustrates the hierarchical window attention mechanism. It begins by constructing an inverse feature pyramid from the input image using a vision transformer (ViT) and a learned upsampling module. The feature maps from different levels of this pyramid are then divided into a uniform grid of windows. These windows, which align across different pyramid levels and serve as regions of interest, undergo RoI alignment to generate spatially consistent sampling features. These sampling features from each level are concatenated together lengthwise and used as the key in the cross-attention block. For each grid location, there is a corresponding learnable query vector. Finally, the output of this cross-attention operation is aggregated and projected into a feature map, which serves as the visual representation for the input image.

read the caption

Figure 4: The flowchart of hierarchical window attention. Feature maps from different levels of the feature pyramid are adaptively RoI-aligned into sampling features and then concatenated along the length axis to serve as the key for the learnable queries.

🔼 This figure compares the performance of a vision-language model on three visual tasks – optical character recognition (OCR), linear probing semantic segmentation (Seg), and fine-grained image classification (Cls) – when using two different up-sampling methods: bilinear interpolation and the proposed Joint Bilateral Upsampling (JBU) module. The JBU module consistently outperforms bilinear interpolation across all tasks, suggesting its ability to preserve more detailed visual information during upsampling. Specifically, JBU leads to a 4% improvement in OCR, 4.7% in segmentation, and 3.8% in classification over bilinear interpolation.

read the caption

Figure 5: Performance on different visual tasks with JBU module and vanilla bilinear interpolation. “OCR” denotes the optical character recognition, “Seg” the Linear probing semantic segmentation, and “Cls” the fine-grained classification on SUB-200.

🔼 This figure presents a qualitative comparison showcasing the enhanced capabilities of LLaVA-UHD v2 in handling high-resolution images involving complex perceptual tasks. It highlights two specific examples (A and B) where LLaVA-UHD v2 demonstrates accurate understanding of image content and question answering compared to other MLLMs like LLaVA-Next, Mini-Gemini, and GPT-4V. These tasks necessitate the model to effectively integrate both fine-grained details (e.g., text in images, small objects) with high-level semantic context.

read the caption

Figure 6: Qualitative comparison of proposed LLaVA-UHD v2 and advanced MLLMs, including LLaVA-Next, Mini-Gemini, and GPT-4V on high-resolution complex perception tasks, which require the integration of both fine-grained visual information and high-level semantic contexts.

🔼 This figure visualizes how different textual tokens activate differently across various levels of the visual feature pyramid. Red circles emphasize noticeable differences in activation patterns between these levels. Essentially, it demonstrates how certain visual features are captured more strongly at different scales (levels of the pyramid). For instance, finer details, such as text, might activate more strongly at higher resolution levels, while broader semantic concepts may have stronger activations at lower resolution levels. This multi-level analysis helps in understanding how the model connects language to visual information at varying granularities.

read the caption

Figure 7: Activation response of specific textual tokens to different visual feature levels. Red circles highlight the obvious difference between levels. (Best viewed in color and zoomed-in)

🔼 Qualitative comparison of LLaVA-UHD v2 with LLaVA-Next, Mini-Gemini, and GPT-4V on high-resolution, complex perceptual tasks requiring both fine-grained visual detail and higher-level semantic understanding, demonstrating superior performance in accurately extracting information from dense text and differentiating targets from similar objects (e.g., identifying the TV program start time, the show date, workout duration, and sale price).

read the caption

Figure 8: Qualitative comparison on high-resolution dense perception task which requires the capabilities of fine-grained details perception.

🔼 LLaVA-UHD v2 and GPT-4V are able to accurately perceive fine-grained details like the beginning time of the television program or small numbers on jerseys and buses, outperforming other models which fail to locate targets or discern them from similar objects.

read the caption

Figure 9: Qualitative comparison on high-resolution fine-grained perception task which requires robust fine-grained visual texture perception capabilities.

🔼 LLaVA-UHD v2 and GPT4V are compared with LLaVA-Next and Mini-Gemini on their spatial perception abilities. The figure presents four examples where models need to extract objects and their spatial relationship in high-resolution images. LLaVA-UHD v2 and GPT4V demonstrate better spatial understanding compared to the other two models.

read the caption

Figure 10: Qualitative comparison on high-resolution spatial perception which necessitates the capabilities of high-level spatial contexts.

🔼 This figure shows a PCA visualization of upsampled features produced by the Joint Bilateral Upsampling (JBU) module applied to a natural scene. Three different upsampling methods are compared: bilinear interpolation, JBU without hierarchical supervision, and JBU with hierarchical supervision. The goal is to demonstrate the effect of these methods on high-resolution (8x the original resolution) feature representation. The figure visually illustrates how hierarchical supervision, when used with the JBU module, helps maintain clearer object boundaries and sharper text details compared to the other two methods. Without hierarchical supervision, the JBU-upsampled features appear somewhat blurry. Bilinear interpolation results in the least clear output.

read the caption

Figure 11: PCA visualization of the up-sampled features by JBU module on nature scene. With hierarchical supervision, the high-resolution features (8×8\times8 ×) could clearly depict object boundary and text appearance. (Best viewed in color and zoomed in)

🔼 This table presents the ablation study results for evaluating the effectiveness of different modules within the LLaVA-UHD v2 model, in particular, the JBU module, High-Resolution Feature Pyramid (HFP) integration, and spatially-consistent token organization. It compares their performance against the baseline LLaVA-UHD model on a variety of visual tasks, including OCR-related tasks, knowledge-based VQA, general vision benchmarks, spatial understanding and high-resolution image perception. The metrics include average accuracy, scores on specific datasets like DocVQA, BenchOCR, and average accuracy on RefCOCO benchmarks. “Δ” represents the overall performance improvement relative to the LLaVA-UHD baseline.

read the caption

Table 2: Ablation studies of modules in our proposed method. “HFP” is the abbreviation of high-resolution feature pyramid. “ΔΔ\Deltaroman_Δ” denotes the overall improvement compared to the baseline. REC reports the average accuracy of RefCOCO/g/+.

🔼 This table compares different methods for constructing a feature pyramid for visual encoding in multimodal large language models (MLLMs). It explores replacing the standard CLIP-ViT with CLIP-ConvNext and directly using its multi-stage feature maps as the hierarchical feature pyramid. It also investigates using deconvolution and bilinear interpolation for feature upsampling, alongside the proposed JBU module, and evaluates their impact on performance across various tasks including general knowledge VQA, OCR-based VQA, and high-resolution image perception.

read the caption

Table 3: Comparison of different methods for feature pyramid construction. “ConvNext” means we replace the CILP-ViT with CLIP-ConvNext [68] as visual encoder and directly use the feature maps from multiple stages as the final hierarchical feature pyramid.

🔼 This table (Table 4) compares different grid sizes for the RoI-Align operation within the hierarchical window attention mechanism, evaluating their impact on performance and efficiency. The ‘Pyramid’ method uses a multi-level grid where the size increases with the feature level (e.g., 2x3 for level 0, 4x6 for level 1), creating a region-level feature pyramid. The ‘Fix’ method uses a fixed 3x3 grid for all feature levels. The table reports training time (Period on 8xA100 GPUs), inference latency (Latency on a single A100 for a 1008x672 image), and GPU memory usage (Memory on 8xA100 GPUs with one image per GPU) during the supervised fine-tuning stage of the MLLM, as well as performance metrics (Average score across tasks, MMEP, GQA, AI2D, VQAC, VQAT, and VQAD) to show the effectiveness of different grid options.

read the caption

Table 4: Comparison of different choice of grid sizes on performance and efficiency. “Pyramid” means the feature grids from different levels form a region-level feature pyramid, e.g.formulae-sequence𝑒𝑔e.g.italic_e . italic_g ., [2×\times×3] for level-0, [4×\times×6] for level-1, [8×\times×12] for leval-2. “Fix” represents all feature maps are pooled into a 3×\times×3 feature grid. We measure the training period on 8×\times×A100s, the latency on an A100 with a 1008×\times×672 image, and the GPU memory on 8×\times×A100s with 1 image per GPU in supervised fine-tune phase.

🔼 This table details the composition of the 858k dataset used for supervised fine-tuning of the Multimodal Large Language Model (MLLM). It lists various data sources, their sizes, and the prompt templates used for response formatting during training.

read the caption

Table 5: Detailed composition of our 858k-mixed dataset.

🔼 This table presents the impact of different feature level choices on model performance and efficiency. It explores configurations using level 0, 2; levels 0, 1, 2; and levels 0, 1, 2, and 3 of the feature pyramid. A configuration without hierarchical supervision is included for levels 0, 1, 2, and 3. The metrics include efficiency measurements like training period and memory usage, as well as performance scores on various VQA benchmarks such as Average, GQA, SQA, REC, VQAC, VQAT, ESTVQA, and MMEP. The table demonstrates how different feature levels and hierarchical supervision affect model performance and resource consumption.

read the caption

Table 6: Comparison of different choices of feature level on performance and efficiency. “HS.”: hierarchical supervision. ESTVQA [101] is a VQA benchmark focusing on scene text recognition.