Multimodal Learning

MAPS solves multimodal scientific problems better by combining multiple agents and Socratic learning.

ETVA evaluates text-to-video alignment via fine-grained question generation and answering.

VidKV: Achieves 1.5x-bit KV cache quantization for VideoLLMs, maintaining performance without retraining.

OThink-MR1 enhances MLLM reasoning via dynamic reinforcement learning, achieving remarkable cross-task generalization!

M3: Gaussian-integrated memory system for multimodal 3D scene understanding with foundation models.

ActVLP: Enhancing VLMs through visual-linguistic guidance for superior action-based decision-making in interactive environments.

UPME: Peer review for MLLMs, minus human bias!

TULIP enhances image-text pretraining by unifying generative data augmentation with contrastive learning, achieving state-of-the-art performance in visual understanding.

BALGRAD mitigates dominant modality bias in vision-language models by reweighting gradients and aligning task directions for balanced learning and improved performance.

Sync your moves! MusicInfuser adapts video diffusion to make models listen and dance to music, preserving style and aligning movement.

MDocAgent: Multi-agent Doc understanding by integrating text and image for better accuracy.

Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMs

Cosmos-Reason1: Physical AI models that reason and act in the real world, bridging the gap between perception and embodied decision-making.

ViSpeak: Enables visual instruction feedback in streaming videos, enhancing human-AI interaction.

SIGHTATION: A BLV-aligned dataset utilizing sighted user feedback to enhance diagram descriptions generated by VLMs, improving accessibility for visually impaired learners.

TVC mitigates visual forgetting in multimodal LLMs, enhancing reasoning by strategically re-introducing and compressing visual information.

MicroVQA: A new benchmark to test visual-question-answering in microscopy-based research.

ADR balances visual-language models by adaptively calibrating long-tail data, boosting LLaVA 1.5 by 4.36% without increasing training data volume.

FreeGrasp: enabling robots to grasp by interpreting instructions and reasoning about object spatial relationships.

DeepPerception enhances MLLMs with cognitive visual perception, achieving superior grounding through knowledge integration & reasoning.