Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion

🔼 Figure 2 illustrates the architecture of Florence-VL, a multimodal large language model. It begins by using Florence-2, a generative vision model, to extract visual features. Crucially, Florence-2 extracts features at multiple ‘depths’ (different levels of abstraction, from low-level details to high-level concepts) and ‘breadths’ (using various prompts to capture different aspects of the image, such as detailed captions, OCR text, and object grounding). These diverse visual features are then fused using a novel Depth-Breadth Fusion (DBFusion) mechanism. The fused features are finally projected into the input space of a large language model (LLM), allowing for effective multimodal understanding. The entire model is first fully pre-trained on image captioning data before undergoing a partial fine-tuning phase using instruction-tuning data.

read the caption

Figure 2: An overview of Florence-VL, which extracts visual features of different depths (levels of feature concepts) and breaths (prompts) from Florence-2, combines them using DBFusion, and project the fused features to an LLM’s input space. Florence-VL is fully pretrained on image captioning data and then partially finetuned on instruction-tuning data.

🔼 This figure visualizes the effectiveness of Florence-2 in capturing various levels of visual information compared to the CLIP model. PCA was applied to image features generated by Florence-2 using three different prompts: Detailed Caption (focuses on overall scene understanding), OCR (focuses on text extraction), and Grounding (highlights spatial relationships between objects). The results show that Florence-2, unlike CLIP, effectively captures fine-grained details such as text within the image. The visualization clearly demonstrates that Florence-2’s multi-faceted visual representations offer a richer, more nuanced understanding of the image than CLIP’s single high-level representation.

read the caption

Figure 3: Visualization of the first three PCA components: we apply PCA to image features generated from Detailed Caption, OCR, and Grounding prompts, excluding the background by setting a threshold on the first PCA component. The image features derived from the Detailed Caption prompt (second column) capture the general context of the image, those from the OCR prompt (third column) focus primarily on text information, and those from the Grounding prompt (fourth column) highlight spatial relationships between objects. Additionally, we visualize the final layer features from OpenAI CLIP (ViT-L/14@336) in the last column, showing that CLIP features often miss certain region-level details, such as text information in many cases.

🔼 This figure presents a comparison of the alignment loss between various vision encoders and a language model. The alignment loss is a measure of how well the visual representations from the encoder align with the textual representations from the language model. Lower alignment loss indicates better alignment. The results show that Florence-2 achieves the lowest alignment loss, indicating the strongest alignment between its visual features and text embeddings compared to other vision encoders like Stable Diffusion, DINOv2, SigLIP, and OpenAI CLIP.

read the caption

Figure 4: We plot the alignment loss for different vision encoders, which clearly shows that Florence-2 vision encoder achieves the lowest alignment loss compared to the other vision encoders, demonstrating the best alignment with text embeddings.

🔼 Figure 5 shows the results of an ablation study on the impact of different visual features on the alignment loss between vision and text representations in a multimodal large language model. Four sets of visual features (using different combinations of depth and breadth of features) are compared: the complete set of features, and the sets that remove either the detailed caption features, OCR features, or grounding features. The graph clearly demonstrates that the combination of all features results in the lowest alignment loss, underscoring the importance of combining features from both different depths (levels of detail) and breadths (various tasks) for optimal cross-modal alignment and model performance.

read the caption

Figure 5: We plot the alignment loss for various feature combinations, removing one feature at a time from different depths and breadths. The results clearly show that our method achieves the lowest alignment loss compared to others, highlighting the importance of all features from different depths and breadths for optimal alignment.

🔼 This table presents the performance comparison of different vision-language models on a range of general multimodal benchmarks. The benchmarks assess various aspects of visual understanding and reasoning capabilities. The models are evaluated based on their accuracy across the benchmarks, providing a comprehensive overview of their strengths and weaknesses. The table includes metrics that quantify the models’ performance on different tasks.

read the caption

(a) Results on general multimodal benchmarks.

🔼 Table 2b presents the performance comparison of various Multimodal Large Language Models (MLLMs) across a range of vision-centric, knowledge-based, and OCR & Chart tasks. It shows the results for different models, including Florence-VL (with both 3B and 8B parameter variants), along with several baselines and other state-of-the-art models. The table details the performance of each model on multiple benchmarks within each category, offering a comprehensive evaluation of their capabilities in diverse multimodal tasks. This is particularly useful for assessing the specific strengths and weaknesses of each model in different domains.

read the caption

(b) Results on Vision centric, Knowledge based, and OCR & Chart benchmarks.

🔼 This table presents a comprehensive evaluation of the Florence-VL model across a diverse range of benchmarks. It’s broken down into four categories: general multimodal benchmarks (assessing overall performance across multiple tasks), vision-centric benchmarks (focus on vision-specific capabilities), knowledge-based benchmarks (testing reasoning and factual understanding), and OCR & Chart benchmarks (evaluating performance on text extraction from images and chart understanding). For each category, the table shows the performance of Florence-VL alongside various baseline models (different sizes and architectures), allowing for direct comparisons and highlighting the model’s strengths and weaknesses in different areas.

read the caption

Table 2: Results on general multimodal benchmarks, Vision centric, Knowledge based, and OCR & Chart benchmarks.

🔼 This table presents the quantitative results of Florence-VL and various baseline models on general multimodal benchmarks. The metrics assess performance across different tasks involving diverse visual and textual inputs. It shows a comparison of the performance of Florence-VL models with varying sizes (3B, 7B, 8B parameters) against other state-of-the-art Multimodal Large Language Models (MLLMs). The benchmarks cover image captioning, question answering, visual reasoning, and other multimodal understanding tasks.

read the caption

(a) Results on general multimodal benchmarks.

🔼 Table 2b presents a breakdown of the performance of various models on three categories of benchmarks: Vision-centric, Knowledge-based, and OCR & Chart. Vision-centric tasks focus on visual understanding and perception. Knowledge-based tasks require reasoning and factual knowledge. OCR & Chart tasks involve extracting information from text in images or charts. The table shows the performance (measured as accuracy) of different models—including the Florence-VL models of varying sizes and several baselines—on each benchmark.

read the caption

(b) Results on Vision centric, Knowledge based, and OCR & Chart benchmarks.

🔼 This table compares the performance of LLaVA 1.5 and Florence-VL (in 3B, 7B, and 8B parameter versions) across a range of multimodal benchmark datasets. The key difference between the models is the vision encoder used: LLaVA 1.5 employs CLIP, while Florence-VL utilizes Florence-2. Importantly, both models were trained using the same training data and underlying large language models (LLMs). The results demonstrate that Florence-VL achieves significantly better performance than LLaVA 1.5, highlighting the advantages of using Florence-2 as the vision encoder.

read the caption

Table 3: We compare LLaVA 1.5 with our model (Florence-VL 3B/7B/8B) across multiple multimodal benchmarks. The key difference between them lies in the vision encoders used (CLIP for LLaVA vs. Florence-2 for our model), while we maintain the same training data and backbone LLMs for both. The results show that our models significantly outperform LLaVA 1.5 with the same training data.

🔼 This table compares the performance of using only lower-level visual features (from the DaViT vision encoder) against using both lower-level and higher-level visual features (from Florence-2). The results show that combining features from different levels (depth) significantly improves the model’s performance across various benchmarks.

read the caption

Table 4: The comparison between keeping only the lower-level feature [𝐕]delimited-[]𝐕[\mathbf{V}][ bold_V ] and our method, which includes both lower- and higher-level features, clearly demonstrates that maintaining both types of features achieves better performance.

🔼 This ablation study investigates the impact of individual high-level visual features extracted by Florence-2 on the overall performance of the Florence-VL model. By systematically removing one high-level feature (Detailed Caption, OCR, Grounding) at a time while keeping other features, the table quantifies the effect on various downstream tasks. The results demonstrate the importance of all three high-level visual features in achieving optimal performance, highlighting the complementary nature of different visual representations.

read the caption

Table 5: Ablation study was conducted by removing one high level image feature at a time, demonstrating that all high-level features are essential for maintaining optimal performance.