FinAudio: A Benchmark for Audio Large Language Models in Financial Applications

Audio Large Language Models (AudioLLMs) have improved audio tasks, but lack benchmarks in finance where audio data (earnings calls) is key for decisions. Financial evaluation suites exist for LLMs in NLP tasks, but there’s a gap: no audio-focused financial LLM or benchmark. Multimodal financial LLMs can’t handle audio data yet. General AudioLLMs have progressed, enabling tasks like ASR, but a financial audio benchmark is missing, limiting research community’s ability to evaluate and improve strategies.

To address this, the paper introduces FINAUDIO, the first AudioLLM benchmark for finance. It defines three tasks: ASR for short/long financial audio, and summarization of long audio. Four open-source datasets were collected, and a new dataset for financial audio summarization was created. Seven AudioLLMs were evaluated, revealing limitations and insights for improvement. The benchmark offers a low-cost, privacy-preserving ASR solution.