Multimodal Learning

Phi-4: Compact Multimodal Language Models via Mixture-of-LoRAs

CLEA: Enhancing task execution in dynamic environments with a closed-loop embodied agent.

Qilin: A multimodal dataset with APP-level user sessions for advancing search and recommendation systems.

HAIC improves MLLMs’ action understanding with high-quality video captions & new benchmark, boosting performance and generation.

UniTok: A unified tokenizer bridging the visual generation and understanding gap via multi-codebook quantization, achieving SOTA in MLLMs.

R2-T2: Boost multimodal MoE performance by re-routing experts in test-time, no retraining needed!

Mobile-Agent-V: Automating mobile tasks using video guidance for efficient, scalable operation, outperforming existing frameworks by 30%.

MMIR: A new benchmark to assess and improve multimodal reasoning models’ ability to detect inconsistencies in real-world content.

KoNET: Evaluating multimodal AI in Korean with edu standards.

SigLIP 2: Multilingual Vision-Language Encoders with Semantic Understanding, Localization, and Dense Features.

CoSyn: Code-guided synth data for scaling text-rich image understanding, achieving SOTA via targeted multimodal data generation!

PC-Agent: A new hierarchical framework that significantly improves complex task automation on PCs by 32%!

InterFeedback: LMMs need better human feedback to enhance AI assistants!

AlphaMaze enhances LLMs’ spatial intelligence via GRPO, achieving 93% accuracy in maze navigation and showing emergent reasoning.

RealSyn: A new, scalable multimodal dataset revolutionizes vision-language learning by effectively using interleaved image-text documents.

mmMamba: a novel framework creates linear-complexity multimodal models via distillation, drastically improving efficiency without sacrificing performance.

Magma: a new foundation model for multimodal AI agents excels at bridging verbal and spatial intelligence, achieving state-of-the-art performance across various tasks, including UI navigation and robo…

video-SALMONN-01: An open-source audio-visual LLM enhances video understanding with a novel reasoning-intensive dataset and the pDPO method, achieving significant accuracy gains.

InfiR: Efficient, small AI models rival larger ones in reasoning, slashing costs and boosting privacy for wider AI use.

HermesFlow seamlessly bridges the understanding-generation gap in MLLMs using a novel Pair-DPO framework and self-play optimization on homologous data, achieving significant performance improvements.