Multimodal Learning

Video-R1: First to explore rule-based RL for video reasoning in MLLMs, enhancing performance on key benchmarks.

UI-R1 enhances GUI agents’ action prediction using reinforcement learning.

Embodied-Reasoner: Integrates visual search, reasoning, and action for interactive tasks, outperforming existing models in embodied environments.

VILBENCH: Vision-Language Process Reward Modeling Suite

UniDisc: a unified multimodal discrete diffusion model for joint text and image generation, surpassing autoregressive models in quality & efficiency!

New metrics and representation enhance 3D talking head realism by focusing on perceptual lip synchronization.

PS3 scales CLIP vision pre-training to 4K resolution with near-constant cost, achieving state-of-the-art performance in multi-modal LLMs.

MLLMs still struggle with spatial reasoning! LEGO-Puzzles benchmark reveals critical deficiencies, paving the way for AI advancement.

HAVEN: A new benchmark to tackle the hallucination issue in video understanding of large multimodal models!

Video SimpleQA: A New Benchmark for Factuality Evaluation in Large Video Language Models.

MetaSpatial: RL for 3D Spatial Reasoning in VLMs

COMP: Continually pre-training Vision Foundation Models for better vision and language alignment and arbitrary size inputs.

AlphaSpace enables robotic actions via semantic tokenization and symbolic reasoning, enhancing spatial intelligence in LLMs.

Vision-R1: Improves LVLMs via vision-guided reinforcement learning, eliminating the need for human feedback and specialized reward models.

VLMs self-improve with text-only training, outperforming vision for human-centered decisions, opening efficient enhancement avenues.

PVChat: Personalize video understanding with one-shot learning, enabling identity-aware video comprehension.

OpenVLThinker: Iteratively refining vision-language models for complex reasoning, bridging the gap to R1-style capabilities.

MAPS solves multimodal scientific problems better by combining multiple agents and Socratic learning.

ETVA evaluates text-to-video alignment via fine-grained question generation and answering.

VidKV: Achieves 1.5x-bit KV cache quantization for VideoLLMs, maintaining performance without retraining.