Multimodal Learning

STEVE: Step-verifying computer-use agent training.

PEBench: A new benchmark for machine unlearning in multimodal language models, enhancing secure multimodal model development.

A comprehensive survey of multimodal chain-of-thought (MCoT) reasoning, bridging the gap in existing literature and fostering innovation towards multimodal AGI.

MPBench: Multimodal benchmark to identify errors in reasoning processes.

CapArena: Detailed image caption benchmark in the LLM era, revealing metric biases and advancing automated evaluation.

Being-0: A humanoid robot agent achieves complex tasks by integrating a vision-language model with modular skills, enhancing efficiency and real-time performance.

VLMs exhibit human-like object categorization, favoring basic levels and mirroring biological/expertise nuances, suggesting learned cognitive behaviors.

HySAC: A hyperbolic framework for safety-aware vision-language models, improving content moderation and interpretability.

V-STaR: A new benchmark to evaluate Video-LLMs in video spatio-temporal reasoning, revealing gaps in current models’ understanding.

D2PO: World modeling enhances embodied task planning by jointly optimizing state prediction and action selection, leading to more efficient execution.

VisualWebInstruct: Scales up multimodal instruction data via web search, enhancing VLMs’ reasoning for complex tasks.

UniGoal: A novel framework for universal zero-shot goal-oriented navigation, outperforming task-specific methods with a unified approach.

LVAS-Agent: Multi-agent system conquers long-video audio synthesis with collaborative dubbing, script, design, & more!

GROVE: Pre-training on large-scale data for grounded video caption generation.

GroundingSuite: A new benchmark that measures complex multi-granular pixel grounding to overcome current dataset limitations and push forward vision-language understanding.

SPIRE: Adds speech to text-only LLMs, maintaining text performance via discretized speech and continued pre-training.

FlowTok: Seamlessly flows across text and image tokens!

Florenz: Scaling laws for systematic generalization via monolingual vision-language models

Cockatiel: Ensembling synthetic & human-preferred training boosts detailed video captioning, setting new SOTA on VDCSCORE.

UniFace: a novel UMM tailored for fine-grained face understanding and generation.