Speech and Audio

FINAUDIO: First benchmark for financial audio LLMs, enhancing financial audio analysis and investment decisions.

Quantization optimizes OpenAI’s Whisper models, balancing model size, speed, and accuracy for diverse applications.

DiffRhythm: Fast & Simple End-to-End Song Generation via Latent Diffusion, creating full songs (4+ mins) with vocal & accompaniment in seconds!

Slam: Train SLMs on one GPU in a day!

SongGen: Single-stage autoregressive transformer for controllable text-to-song generation, simplifying the process and improving control.

FocalCodec: a single codebook, low-bitrate speech codec using focal modulation, achieves competitive performance in speech resynthesis and voice conversion.

Emilia-Pipe and its resulting datasets, Emilia and Emilia-Large, offer the largest open-source, multilingual speech corpus, enabling more natural and spontaneous AI speech generation.

HiFi-SR: A unified generative network achieves high-fidelity speech super-resolution, outperforming existing methods by seamlessly integrating transformer and convolutional components for end-to-end a…

XMusic: A new framework generates high-quality, emotionally controllable symbolic music from various prompts (images, videos, text, tags, humming).

Whisper-GPT, a hybrid audio LLM, improves music/speech generation by combining audio waveforms and text.