Multimodal Generation

UniDisc: a unified multimodal discrete diffusion model for joint text and image generation, surpassing autoregressive models in quality & efficiency!

Sync your moves! MusicInfuser adapts video diffusion to make models listen and dance to music, preserving style and aligning movement.

FlowTok: Seamlessly flows across text and image tokens!

UniFace: a novel UMM tailored for fine-grained face understanding and generation.

Motion Anything: control human motion generation with multimodal conditions like text and music.

SURVEYFORGE automates survey generation, improving quality and evaluation.

VideoUFO: A new user-focused, million-scale dataset that improves text-to-video generation by aligning training data with real user interests and preferences!

MMAudio achieves state-of-the-art video-to-audio synthesis by jointly training on audio-visual and text-audio data, enabling high-quality, semantically and temporally aligned audio generation.

VMB generates music from videos, images, and text, using description and retrieval bridges to improve quality and controllability.

LatentLM: a novel multimodal model unifying discrete & continuous data via next-token diffusion, surpassing existing methods in performance & scalability across various tasks.

OmniFlow: a novel generative model masters any-to-any multi-modal generation, outperforming existing models and offering flexible control!

SmoothCache: A universal technique boosts Diffusion Transformer inference speed by 8-71% across modalities, without sacrificing quality!