Multimodal Learning

SegAgent: Improves MLLMs’ pixel understanding by mimicking human annotation, enabling mask refinement without altering output space.

Introducing HumanRef, a new dataset & RexSeek, a multimodal LLM, to improve human-centric referring tasks by addressing limitations of existing methods.

OmniMamba: Efficient multimodal understanding and generation via SSMs, trained on 2M image-text pairs.

GTR: Prevents thought collapse in RL-based VLM agents by process guidance, enhancing performance in complex visual reasoning tasks.

RFLAV: A novel rolling flow matching model for infinite audio-video generation with high quality, synchronization, and temporal coherence.

VidDiff: Identify subtle action differences in videos for coaching and skill learning.

Image data during pre-training can boost Vision-Language Model (VLM) performance, especially when introduced later in the process.

Motion Anything: control human motion generation with multimodal conditions like text and music.

VisualSimpleQA: A new benchmark for fine-grained evaluation of visual and linguistic modules in fact-seeking LVLMs.

ProJudge: MLLM judges’ benchmark for sci-reasoning & instruction-tuning data to boost performance!

DiffCLIP: Enhancing CLIP models by integrating differential attention, achieving superior performance with minimal overhead.

ARMOR: Empowers MLLMs with interleaved multimodal generation via asymmetric synergy, using limited resources.

Llama-MTSK: AVSR via Matryoshka LLMs, adapting to computational limits without sacrificing accuracy!

UNIFIEDREWARD: A unified reward model that enhances multimodal understanding and generation!

R1-Omni: RLVR enhances multimodal emotion recognition, boosting reasoning and generalization.

SURVEYFORGE automates survey generation, improving quality and evaluation.

EgoLife: Ultra-long egocentric dataset & benchmark enabling AI assistants to understand and enhance daily life. Datasets and models released!

VLMs often disproportionately trust text over visual data, leading to performance drops and safety concerns.

Visual-RFT: Enhance LVLMs’ visual reasoning via reinforcement learning with verifiable rewards, achieving strong performance with limited data.

VideoUFO: A new user-focused, million-scale dataset that improves text-to-video generation by aligning training data with real user interests and preferences!