Multimodal Generation

MMAudio achieves state-of-the-art video-to-audio synthesis by jointly training on audio-visual and text-audio data, enabling high-quality, semantically and temporally aligned audio generation.

VMB generates music from videos, images, and text, using description and retrieval bridges to improve quality and controllability.

LatentLM: a novel multimodal model unifying discrete & continuous data via next-token diffusion, surpassing existing methods in performance & scalability across various tasks.

OmniFlow: a novel generative model masters any-to-any multi-modal generation, outperforming existing models and offering flexible control!

SmoothCache: A universal technique boosts Diffusion Transformer inference speed by 8-71% across modalities, without sacrificing quality!