Audio-Visual Learning

One model to rule them all! This paper introduces Unified Speech Recognition (USR), a single model trained for auditory, visual, and audiovisual speech recognition, achieving state-of-the-art results …

UniAudio 1.5 uses a novel LLM-driven audio codec to enable frozen LLMs to perform various audio tasks with just a few examples, opening new avenues for efficient few-shot cross-modal learning.

VATT: Text-guided video-to-audio generation, enabling refined audio control via text prompts and improved compatibility.

AVMoE: a novel parameter-efficient transfer learning approach for audio-visual learning, dynamically allocates expert models (unimodal and cross-modal adapters) based on task demands, achieving superi…

LMAC-ZS: First decoder-based method for explaining zero-shot audio classifiers, ensuring transparency and trustworthiness in AI.

ELSA: a new model that learns spatially aware language and audio embeddings, achieving state-of-the-art performance in semantic retrieval and 3D sound source localization.

FRIEREN: a novel video-to-audio generation network using rectified flow matching achieves state-of-the-art performance by improving audio quality, temporal alignment, and generation efficiency.

DARNet: a dual attention network for auditory attention detection surpasses current state-of-the-art models, especially in short decision windows, achieving this with a 91% reduction in parameters.

ContAV-Sep: a novel approach to continual audio-visual sound separation, effectively mitigating catastrophic forgetting and improving model adaptability by preserving cross-modal semantic similarity a…

AV-GS: A novel Audio-Visual Gaussian Splatting model, uses geometry and material-aware priors to efficiently synthesize realistic binaural audio from a single audio source.

AV-Cloud: Real-time, high-quality 3D spatial audio rendering synced with visuals, bypassing pre-rendered images for immersive virtual experiences.

AVAgent uses an LLM-driven workflow to intelligently align audio and visual data, resulting in improved AV joint representations and state-of-the-art performance on various downstream tasks.

A single model tackles diverse audiovisual generation tasks using a novel Mixture of Noise Levels approach, resulting in temporally consistent and high-quality outputs.