Multimodal Learning

3DGraphLLM boosts 3D scene understanding by cleverly merging semantic graphs and LLMs, enabling more accurate scene descriptions and outperforming existing methods.

PC Agent: While you sleep, AI works! This AI system uses human cognition transfer to perform complex digital tasks, exceeding the capabilities of existing digital agents by efficiently learning from h…

M-STAR: a novel self-evolving training framework significantly boosts multimodal reasoning in large models without human annotation, achieving state-of-the-art results.

MMAudio achieves state-of-the-art video-to-audio synthesis by jointly training on audio-visual and text-audio data, enabling high-quality, semantically and temporally aligned audio generation.

AR-MCTS: a novel framework boosting multimodal large language model reasoning by actively retrieving key supporting evidence and using Monte Carlo Tree Search for improved path selection and verificat…

MegaPairs synthesizes 26M+ high-quality multimodal retrieval training examples, enabling state-of-the-art zero-shot performance and surpassing existing methods trained on 70x more data.

CrossFlow: Directly evolve any modality to another using flow matching, achieving state-of-the-art results across various tasks!

LLaVA-UHD v2 enhances MLLMs by integrating high-resolution visual details using a hierarchical window transformer.

A comprehensive survey of GUI agents, categorizing benchmarks, architectures, training methods, and open challenges, providing a unified framework for researchers.

Enhance image captions significantly with DCE, a novel engine leveraging visual specialists to generate comprehensive, detailed descriptions surpassing LMM and human-annotated captions.

New benchmark reveals how well AI understands and meets real-world human needs.

Apollo LMMs achieve SOTA on video understanding tasks by exploring and optimizing the design and training of video-LMMs.

SynerGen-VL: A simpler, more powerful unified MLLM for image understanding and generation.

OLA-VLM boosts multimodal LLMs’ visual understanding by distilling knowledge from specialized visual encoders into the LLM’s internal representations during pretraining, achieving significant performa…

VMB generates music from videos, images, and text, using description and retrieval bridges to improve quality and controllability.

Lyra: An efficient, speech-centric framework for omni-cognition, achieving state-of-the-art results across various modalities while being highly efficient.

GenEx generates explorable 3D worlds from a single image, enabling embodied AI agents to explore and learn.

LatentLM: a novel multimodal model unifying discrete & continuous data via next-token diffusion, surpassing existing methods in performance & scalability across various tasks.

BiMediX2, a bilingual medical expert LMM excels in diverse medical modalities.

ILLUME: A unified multi-modal LLM efficiently integrates visual understanding & generation, achieving competitive performance with significantly less data.