On the Compositional Generalization of Multimodal LLMs for Medical Imaging

🔼 This figure illustrates the creation of the Med-MAT dataset. It starts with 106 individual medical image datasets. These datasets are categorized and grouped based on their MAT-Triplet (Modality, Anatomical Area, and Task), resulting in 53 subsets. Each subset consists of various modalities, anatomical areas, and medical tasks. The QA pairs construction is shown, where image-label pairs are converted into question-answer pairs suitable for MLLM training. This process ensures each sample in Med-MAT is clearly defined by its MAT-Triplet, enabling research on compositional generalization.

read the caption

Figure 2: The process of integrating a vast amount of labeled medical image data to create Med-MAT.

🔼 This figure illustrates the process of converting image-label pairs in Med-MAT into a visual question-answering (VQA) format suitable for training and evaluating multimodal large language models (MLLMs). The process involves assigning multiple instructions to each subset of images, converting each image-label pair into a single-choice question with four options, and randomly selecting distractor options from other labels within the subset. The integration of the ImageWikiQA dataset helps mitigate potential evaluation biases from varying option counts. The figure provides a detailed breakdown of the transformation from raw medical images and captions to structured questions and answers, which are essential for the MLLM training and testing.

read the caption

Figure 3: The QA formatting process of Med-MAT.

🔼 Figure 4 presents a comparative analysis of the generalization performance of several multimodal large language models (MLLMs) on unseen medical image data. The models were trained using different strategies, and their accuracy on a target dataset is shown. Specifically, the figure displays results for models trained on all related datasets, all unrelated datasets, all related datasets excluding those that share a Modality, Anatomical Area, or Task with the target dataset (to disrupt compositional generalization, or CG), and models trained on all available datasets. The target data itself was excluded from training to assess genuine generalization ability. This visual comparison helps illustrate the impact of compositional generalization on the model’s capacity to generalize.

read the caption

Figure 4: Accuracy results on the Target dataset for various models. ’All Related/Unrelated’ models are trained on all the related or unrelated datasets of the Target Data. ’w/o Modality/Area/Task’ are trained on All Related datasets but omit those sharing the same element as the Target Data, to intentionally disrupt CG. ’All Data’ uses all available training sets. (Note: The Target Data is excluded from training to observe generalization.)

🔼 This table presents the accuracy of different models on an Out-of-Distribution (OOD) dataset. The OOD dataset consists of medical images not seen during the model’s training phase. The table compares the performance of a baseline model, a single-task training model, and a multi-task training model. Bold text highlights the best accuracy scores achieved for each task by any of the models, indicating which model performed best on each OOD dataset subset.

read the caption

Table 2: Accuracy of different models on Out-Of-Distribution Dataset. Bold highlights the best scores.

🔼 Table 3 presents the results of a controlled experiment evaluating compositional generalization (CG) in a medical image classification task. The table shows the accuracy of a multimodal large language model (MLLM) on various target datasets (unseen during training). Each row represents a different target dataset and its associated training set. The ‘Related Combination’ column specifies the training set of datasets containing images with similar characteristics (same Modality, Anatomical Area, and Task). The ‘Target Subset’ column identifies the dataset tested. Three accuracy scores are provided: Baseline (no training), Baseline+ (trained on randomly selected unrelated datasets), and Trained (trained on the ‘Related Combination’ dataset). The green sections highlight successful generalization (accuracy improved by using related data), while red sections indicate unsuccessful generalization (no improvement or decrease in accuracy by using related data). The table is divided into four sections that represent different combinations of fixed elements (modality, area, or task) within the MAT-Triplet, allowing for analysis of how compositional generalization varies based on the combination of similar data.

read the caption

Table 3: Generalization results on classification datasets: 'Related Combination' is the training set, 'Target Subset' is the goal. Baseline, Baseline+, and Trained represent the model’s accuracy without training, trained on randomly sampled unrelated data, and trained on related data, respectively. Green section indicates successful generalization, while red section denotes failure. The 4 segmented areas represent different Direction Types: fixed modality, fixed area, fixed task, and modality-area paired combinations.

🔼 This table presents the results of a controlled experiment designed to investigate compositional generalization (CG) in multi-modal large language models (MLLMs). Three datasets were selected, each providing different combinations of MAT-Triplet elements (Modality, Anatomical area, and Task). The experiment tested whether the model could generalize to a target subset (a new combination of MAT-Triplet elements) by training only on related combinations of MAT-Triplet elements. The table shows the baseline performance (no training), and the accuracy of the model trained on related combinations. Green indicates successful generalization (the model correctly predicted the target task using only related training data), while red indicates failure.

read the caption

Table 4: Generalization results from 3 datasets providing different elements of MAT-Triplet (RQ 3). 'Related Combination' is the training set, 'Target Subset' is the goal. Baseline, and Trained represent the model’s accuracy without training and trained on Related data, respectively. Green section indicates successful generalization, while red section denotes failure.

🔼 Table 5 presents the results of using the NEXT-Chat model to assess compositional generalization (CG). The experiment focuses on whether training with a combination of detection and classification datasets improves the model’s ability to generalize to unseen classification tasks. The table shows different combinations of related datasets used for training (‘Related Combination’), the target dataset tested (‘Target Subset’), and the model’s accuracy with no training (‘Baseline’), and after training on related data (‘Trained’). The results are categorized into four ‘Direction Types’ based on how the related and target datasets were related: fixed modality, fixed area, fixed task, and paired modality-area combinations. Green highlights successful generalization, while red indicates failure. This helps understand how different relationships between training and target datasets impact generalization performance.

read the caption

Table 5: Result of NEXT-Chat on CG by using detection and classification tasks to generalize classification Target dataset. Generalization results on classification datasets: 'Related Combination' is the training set, 'Target Subset' is the goal. Baseline and Trained represent the model’s accuracy without training and trained on related data, respectively. Green section indicates successful generalization, while red section denotes failure. The 4 segmented areas represent different Direction Types: fixed modality, fixed area, and modality-area paired combinations.

🔼 This table presents the results of using MiniGPT-v2, a multimodal large language model, to perform a compositional generalization task. The model was trained on various combinations of datasets (‘Related Combination’) with a shared MAT-Triplet (Modality, Anatomical Area, Task) to predict the accuracy on a target dataset (‘Target Subset’) with unseen combinations of these elements. ‘Baseline’ represents the model’s accuracy without any training on related datasets, while ‘Trained’ reflects the accuracy after training with related datasets. The table highlights successful (‘Green’) and unsuccessful (‘Red’) generalization, categorized by three ‘Direction Types’ that reflect variations in dataset selection: fixed modality, fixed area, and modality-area paired combinations. This helps understand how the model’s ability to generalize depends on the relationships between the training and test data based on shared MAT-Triplets.

read the caption

Table 6: Result of MiniGPT-v2 on CG by using detection and classification tasks to generalize classification Target dataset. Generalization results on classification datasets: 'Related Combination' is the training set, 'Target Subset' is the goal. Baseline and Trained represent the model’s accuracy without training and trained on related data, respectively. Green section indicates successful generalization, while red section denotes failure. The 3 segmented areas represent different Direction Types: fixed modality, fixed area, and modality-area paired combinations.

🔼 Table 7 presents the results of using the Qwen2-VL model on a subset of classification datasets from the Med-MAT dataset. The table showcases the model’s performance in generalizing to unseen data based on training with related and unrelated data. Green highlights successful generalization, while red denotes failure. Each row represents a pair of related and target datasets, indicating which combinations successfully promote generalization to the target dataset.

read the caption

Table 7: Result of Qwen2-VL on selected classification datasets in Med-MAT. Green section indicates successful generalization, while red section denotes failure.

🔼 Table 8 presents the results of using the Llama-3.2-Vision model on a subset of classification datasets from the Med-MAT dataset. The table shows the model’s performance on unseen data (‘Target Subset’) after being trained on related datasets (‘Related Combination’). The results are categorized to show successful generalization (green) or failure (red). The categories represent different types of relationships between the training and target datasets, based on fixed modalities, anatomical areas, tasks, or combinations of these factors.

read the caption

Table 8: Result of Llama-3.2-Vision on selected classification datasets in Med-MAT. Green section indicates successful generalization, while red section denotes failure.

🔼 Table 9 details the composition of the Med-MAT dataset, showing how it’s divided into subsets. Each subset contains medical images categorized by three elements: Modality (e.g., CT scan, MRI), Anatomical Area (e.g., lung, brain), and Task (e.g., cancer detection, disease classification). The table lists each subset, specifying its modality, anatomical area, task, and the number of datasets included. It also uses color-coding to distinguish between classification and detection tasks within the subsets. Abbreviations for modalities (e.g., Co for Colposcopy, CT for Computed Tomography, etc.) are provided in the caption to aid understanding.

read the caption

Table 9: The details of subset. In particular, Co stands for Colposcopy, CT represents Computed Tomography, DP refers to Digital Photography, FP is for Fundus Photography, MRI denotes Magnetic Resonance Imaging, OCT signifies Optical Coherence Tomography, Der refers to Dermoscopy, End stands for Endoscopy, Mic indicates Microscopy Images, and US represents Ultrasound. The blue section represents the classification dataset and the green section represents the detection

🔼 Table 10 provides detailed information on the 109 medical datasets used in the study. For each dataset, it lists the dataset name, a brief description of its contents (e.g., type of medical images, specific diseases or conditions), and the corresponding citation from the literature where the dataset is originally described. This table is crucial for understanding the breadth and diversity of the data used to train and evaluate the multimodal large language models (MLLMs) in the paper, particularly concerning compositional generalization.

read the caption

Table 10: The details of the medical datasets are provided

🔼 Table 11 provides a continuation of the dataset descriptions started in Table 10. It lists additional medical image datasets used in the study, detailing their names, descriptions of the medical tasks involved (e.g., classification, segmentation, detection), and the citation for each dataset’s source. The table is crucial for understanding the breadth and diversity of data used to evaluate the model’s capabilities and generalization performance.

read the caption

Table 11: Continued from Table 10.