Multimodal Learning

RIG: Synergizes reasoning and imagination in an end-to-end generalist policy for embodied agents, improving sample efficiency and generalization.

KOFFVQA: Objectively evaluates Korean VLMs with a new free-form VQA benchmark, improving evaluation reliability via detailed grading criteria.

MoCha: Movie-Grade Talking Character Synthesis!

Video-R1: First to explore rule-based RL for video reasoning in MLLMs, enhancing performance on key benchmarks.

UI-R1 enhances GUI agents’ action prediction using reinforcement learning.

Embodied-Reasoner: Integrates visual search, reasoning, and action for interactive tasks, outperforming existing models in embodied environments.

VILBENCH: Vision-Language Process Reward Modeling Suite

UniDisc: a unified multimodal discrete diffusion model for joint text and image generation, surpassing autoregressive models in quality & efficiency!

New metrics and representation enhance 3D talking head realism by focusing on perceptual lip synchronization.

PS3 scales CLIP vision pre-training to 4K resolution with near-constant cost, achieving state-of-the-art performance in multi-modal LLMs.

MLLMs still struggle with spatial reasoning! LEGO-Puzzles benchmark reveals critical deficiencies, paving the way for AI advancement.

HAVEN: A new benchmark to tackle the hallucination issue in video understanding of large multimodal models!

Video SimpleQA: A New Benchmark for Factuality Evaluation in Large Video Language Models.

MetaSpatial: RL for 3D Spatial Reasoning in VLMs

COMP: Continually pre-training Vision Foundation Models for better vision and language alignment and arbitrary size inputs.

AlphaSpace enables robotic actions via semantic tokenization and symbolic reasoning, enhancing spatial intelligence in LLMs.

Vision-R1: Improves LVLMs via vision-guided reinforcement learning, eliminating the need for human feedback and specialized reward models.

VLMs self-improve with text-only training, outperforming vision for human-centered decisions, opening efficient enhancement avenues.

PVChat: Personalize video understanding with one-shot learning, enabling identity-aware video comprehension.

OpenVLThinker: Iteratively refining vision-language models for complex reasoning, bridging the gap to R1-style capabilities.