BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities

↗ arXiv ↗ Hugging Face ↗ Papers with Code

TL;DR

#

Key Takeaways

#

Why does it matter?

#

BiMediX2’s bilingual, multimodal approach is crucial for researchers exploring inclusive healthcare solutions. It offers a robust framework for developing similar models in other languages, bridging healthcare access gaps. The extensive BiMed-V dataset provides valuable resources for multimodal medical research. Finally, BiMed-MBench sets a new standard for evaluating bilingual medical LMMs, fostering further advancements in the field.

Visual Insights

#

🔼 This radar chart compares the performance of several large multimodal models (LMMs) on the BiMed-MBench, a bilingual (Arabic-English) medical benchmark. The models evaluated include LLaVA-pp, LLaVA-Med, BiMediX2, Dragonfly-Med, MiniGPT-Med, and BiomedGPT. Performance is assessed across different medical image categories: Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Chest X-Ray (CXR), Histology, and Gross pathology, in both English and Arabic. Each axis of the chart represents a specific category, and the model’s score on that category determines its distance from the center. This visualization allows for easy comparison of model performance across different modalities and languages.

read the caption

Figure 1: Model Performance Comparison on BiMed-MBench: These comparisons are made across different categories, including CT, MRI, CXR, Histology, Gross, and their Arabic counterparts (CT_ar, MRI_ar, CXR_ar, Histology_ar, Gross_ar). The models compared are LLaVA-pp, LLaVA-Med, BiMediX2, Dragonfly-Med, MiniGPT-Med, and BiomedGPT. Each axis represents the performance score in a specific category, allowing for a visual comparison of how each model performs in bilingual medical contexts.

🔼 This table provides a comparison of recent Medical Large Language Models (LLMs) and Vision-Language Models (VLMs), highlighting their capabilities across various medical tasks and modalities. The table uses abbreviations like MTC for Multi-turn conversation, RS for Report Summarization, and others as explained in the caption. It also indicates whether these models offer a Unified Model (UM) approach (single checkpoint for all tasks) or a combined LLM+VLM architecture, and whether they have Bilingual (Arabic) capabilities.

read the caption

Table 1: Comparison of Recent Medical LLMs and VLMs. Abbreviations: MTC (Multi-turn conversation), RS (Report Summarization), RG (Report Generation), Rad (Radiology), Oph (Ophthalmology), Path (Pathology), Micro (Microscopic), UM (Unified Model: Single model checkpoint for all downstream tasks), LLM+VLM (Unified LLM + VLM), Bil (Ar) (Bilingual Arabic capabilities).

In-depth insights

#

Bilingual Medical LMM

#

BiMediX2, a novel bilingual Medical Large Multimodal Model (LMM), addresses the critical need for inclusive healthcare solutions. By supporting both Arabic and English, it bridges a significant gap in medical AI accessibility. This bilingual capability allows diverse populations to engage with advanced medical information and diagnostics, promoting health equity. BiMediX2 leverages a unified architecture, integrating text and visual modalities for comprehensive medical understanding. Trained on a massive bilingual dataset, BiMed-V, it excels in tasks like medical image analysis and multi-turn conversations. The development of the first bilingual GPT-40 based medical LMM benchmark, BiMed-MBench, facilitates rigorous evaluation and comparison with existing models, demonstrating BiMediX2’s superior performance. While challenges like hallucinations remain, BiMediX2 represents a substantial advancement, paving the way for more inclusive and effective global healthcare.

BiMediX2 Architecture

#

BiMediX2’s architecture effectively integrates textual and visual data for enhanced medical analysis. A Vision Encoder processes medical images, creating visual embeddings. Simultaneously, text inputs are converted into textual embeddings using a tokenizer and the LLaMA 3.1 language model. A Projector aligns these two modalities, mapping visual features to corresponding textual concepts. This unified approach facilitates tasks like image captioning and visual question answering in a bilingual (Arabic-English) context. LoRA adapters enable efficient fine-tuning of the language model while preserving computational resources. This design promotes multi-turn conversations about medical images, fostering a more interactive and informative diagnostic experience. The project’s innovative bilingual dataset and benchmark further enhance its ability to provide inclusive and comprehensive healthcare solutions.

BiMed-V Dataset

#

The BiMed-V dataset is a multilingual, multimodal medical instruction set. Its 1.6M samples enhance medical image-text alignment and multimodal understanding. It leverages existing datasets like PMC-OA, Rad-VQA, Path-VQA, and SLAKE, supplemented with custom data and repurposed LLaVA-Med examples. Bilingual support is a key feature, with 163k Arabic samples generated via GPT-40 translation and expert validation. This hybrid approach minimizes reliance on human experts while ensuring quality. The inclusion of BiMediXv1’s text-based clinical data strengthens language understanding. BiMed-V enables advanced medical image-text alignment and conversational applications, addressing the need for inclusive healthcare solutions.

Multimodal Eval

#

BiMediX2, a bilingual (Arabic-English) medical Large Multimodal Model (LMM), undergoes a multimodal evaluation to assess its proficiency in processing and understanding medical images along with textual queries. The evaluation leverages various benchmarks and datasets including BiMed-MBench, a novel bilingual medical benchmark, and established VQA datasets like Rad-VQA, SLAKE, and Path-VQA. Performance metrics include accuracy, recall, F1-score, and BLEU scores for tasks such as Visual Question Answering, report generation (using MIMIC-CXR), and report summarization (using MIMIC-III). The model’s ability to interpret diverse imaging modalities like X-rays, CT scans, MRIs, and histology slides, coupled with its bilingual capabilities, is rigorously tested, providing a comprehensive assessment of its potential in real-world medical applications. The robust evaluation framework underscores the emphasis on accuracy, clinical relevance, and language proficiency. This helps in creating more inclusive and effective medical AI solutions.

Arabic & MedImg Focus

#

BiMediX2’s Arabic focus addresses a critical gap in medical AI, serving Arabic-speaking populations. This inclusivity broadens access to advanced medical insights and fosters more equitable healthcare. The model’s training on a large, bilingual dataset, including translated and expert-verified medical texts and images, enhances its understanding of Arabic medical terminology and nuances. BiMediX2’s strength in medical image analysis, combined with its bilingual capabilities, empowers healthcare professionals in Arabic-speaking regions to leverage cutting-edge technology for improved diagnosis, treatment planning, and patient care. This inclusivity in medical AI represents a significant step toward reducing health disparities and promoting global health equity. Further research will explore regional dialects and cultural contexts to enhance BiMediX2’s sensitivity and relevance within diverse Arabic-speaking communities.

More visual insights

#

More on figures

🔼 BiMediX2 processes medical images through a Vision Encoder and aligns them with text using a Projector. Text is tokenized and input to Llama 3.1, which generates responses in the prompted language. The language model is trained using LoRA adapters, and the projector is fine-tuned for medical image-text alignment. An English data corpus is translated to Arabic using GPT-40 and verified by medical experts, facilitating bilingual training and benchmarking.

read the caption

Figure 2: BiMediX2: Overall Architecture Our model is designed for medical image analysis and bilingual multi-turn conversations. Medical images are processed through a Vision Encoder and aligned with a Projector, while the text inputs are tokenized using the default tokenizer. The resulting tokens are then passed into the language model (Meta Llama 3.1) to generate responses in the prompted language. We only train the language model using LoRA adapters, while the projector is finetuned for medical image-text alignment. A robust data generation framework translates an English data corpus into Arabic using GPT-4o, with verification by a medical expert to ensure accurate and contextually appropriate translations. This approach supports effective training and benchmarking in a bilingual context.

🔼 This figure presents a comparison of the state-of-the-art models in Clinical Large Language Model (LLM) benchmarks. Each bar represents a different model and its height represents the average performance across various clinical datasets (Cli-KG, C-Bio, C-Med, Med-Gen, Pro-Med, Ana, MedMCQA, MedQA, USMLE, PubmedQA). BiMediX2 models (4B, 8B, and 70B) are compared against existing LLM models (BioMedGPT, LLaVA-Med, Dragonfly-Med, GPT 3.5 & 4, Meditron, Llama3-Med42, OpenBioLLM, Llama 3.1). Overall, the results show that larger models generally exhibit higher average performance and the proposed models demonstrate competitive performance against existing models on this benchmark.

read the caption

Figure 3: State of the art comparison of models in Clinical LLM Benchmarks

🔼 This figure compares the performance of several large language models (LLMs), including BiMediX2, on the UPHILL OpenQA benchmark. The UPHILL benchmark tests the factual accuracy of LLMs in handling health-related queries, specifically focusing on the models’ ability to correctly refute false medical claims. The x-axis shows the model names, and the y-axis represents the overall factual accuracy percentage achieved by each model on the benchmark.

read the caption

Figure 4: Performance comparison on UPHILL OpenQA (Kaur et al. (2023)), assessing the model’s ability to address false medical claims at different presupposition levels.

🔼 Figure 5 showcases BiMediX2’s capabilities in analyzing medical images within a conversational context. The top part presents a conversation about a sagittal CT scan of the lumbar spine, where BiMediX2 identifies the scan type, describes the anatomy, and pinpoints a fracture in the L4 vertebra, explaining potential causes. The bottom section features a color Doppler ultrasound of the left ovary. Here, the model explains the technique, identifies the organ, and notes a potential left ovarian cyst with a solid component, highlighting the need for further evaluation. These examples illustrate BiMediX2’s ability to interpret complex medical images and engage in informative medical conversations.

read the caption

Figure 5: Qualitative Examples of our BiMediX2 for Medical Image Understanding in a Conversational Context.

Full paper

#