TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization

🔼 Figure 2 illustrates the performance of two training methods (online and offline) for a text-to-audio model across multiple iterations. The y-axis shows the CLAP score (a measure of audio quality and relevance to the input text) and KL divergence (a measure of the difference between the generated audio distribution and the target distribution). The x-axis represents the training iteration. The offline training method demonstrates rapid initial improvement, peaking around the second iteration, but then shows diminishing returns with a subsequent decline in CLAP score and an increase in KL divergence. In contrast, the online training method exhibits sustained improvement, with the CLAP score increasing steadily until the fourth iteration and a simultaneous decrease in KL divergence, suggesting a more stable and effective training process.

read the caption

Figure 2: The trajectory of CLAP score and KL divergence across the training iterations. This plot shows the stark difference between online and offline training. Offline training clearly peaks early, by the second iteration, indicated by the peaking CLAP score and increasing KL. In contrast, the CLAP score of online training continues to increase until iteration 4, while the KL divergence has a clear downward trend throughout.

🔼 Figure 3 illustrates the training dynamics of two different loss functions: ℒDPO-FM and ℒCRPO, used for preference optimization in a text-to-audio model. The plot shows how the ‘winning’ and ’losing’ losses (representing the difference in model preference between better and worse audio samples) change over five iterations of training. Notably, both the winning and losing losses increase in magnitude with each training iteration. Importantly, the difference between the winning and losing losses (their margin) also increases, suggesting successful preference alignment with each iteration. This visualization underscores the iterative nature of the proposed preference optimization process and how the chosen loss function affects the training dynamics, leading to improved audio quality.

read the caption

Figure 3: Winning and Losing losses of ℒDPO-FMsubscriptℒDPO-FM\mathcal{L}_{\text{DPO-FM}}caligraphic_L start_POSTSUBSCRIPT DPO-FM end_POSTSUBSCRIPT and ℒCRPOsubscriptℒCRPO\mathcal{L}_{\text{CRPO}}caligraphic_L start_POSTSUBSCRIPT CRPO end_POSTSUBSCRIPT at each iteration. Winning and Losing losses increase each iteration, as well as their margin.

🔼 The figure shows the trajectory of CLAP score and KL divergence across training iterations. The CLAP score is a metric measuring the alignment between generated audio and text descriptions. KL divergence measures the difference between the generated audio distribution and the target distribution. The plot illustrates the performance of two training methods: online CRPO (with new data generated each iteration) and offline CRPO (using the same data repeatedly). The online CRPO shows a steady increase in CLAP scores and a decrease in KL divergence across iterations, indicating improved performance and better alignment. The offline CRPO, in contrast, shows early peaking of the CLAP score and an increase in KL divergence, demonstrating performance saturation and degradation after a few iterations. This highlights the importance of online data generation in the CRPO framework for consistently improving TTA model alignment.

read the caption

(a) CLAPscorescore{}_{\text{score}}start_FLOATSUBSCRIPT score end_FLOATSUBSCRIPT

🔼 Figure 2 shows the trends of two key metrics, the KL divergence and CLAP score, across five iterations of the CLAP-Ranked Preference Optimization (CRPO) process. The blue line represents the CRPO approach with online data generation at each iteration, whereas the red line represents CRPO without online data generation. The KL divergence is a measure of the difference between the generated audio distribution and the true audio distribution, while the CLAP score measures the alignment between the generated audio and text descriptions. As the iterations proceed, the online CRPO consistently exhibits a decreasing KL divergence, signifying improved alignment with the true distribution, while the CLAP score steadily increases, indicating better adherence to the provided textual prompts. In contrast, the offline CRPO approach demonstrates a more erratic behavior, showing early improvement followed by degradation in both metrics after the second iteration. This highlights the importance of incorporating online data generation in CRPO to prevent performance saturation and ensure consistent progress.

read the caption

(b) FDopenl3openl3{}_{\text{openl3}}start_FLOATSUBSCRIPT openl3 end_FLOATSUBSCRIPT

🔼 This table presents a comparison of various text-to-audio models’ performance on audio clips containing multiple events. The models compared are AudioLDM 2-large, Stable Audio Open, Tango 2, TANGOFLUX-base, and TANGOFLUX. For each model, the table lists the number of parameters, the duration of the generated audio, the Fréchet distance (FDopenl3), Kullback-Leibler divergence (KLpasst), CLAP score, and Inception Score (IS). Lower FDopenl3 and KLpasst scores, and higher CLAP and IS scores indicate better performance. This allows for a direct comparison of how well each model handles complex audio scenarios with multiple events.

read the caption

Table 2: Comparison of text-to-audio models on multi-event inputs.

🔼 This table compares the performance of three different preference datasets used to fine-tune a text-to-audio model. The datasets are BATON, Audio-Alpaca, and the novel CRPO dataset proposed in this paper. The comparison is based on three metrics: FDopenl3 (Fréchet distance for measuring the similarity of audio distributions), KLpasst (Kullback-Leibler divergence to quantify the difference between generated and reference audio label distributions), and CLAP score (measuring the alignment of generated audio with text descriptions). Lower FDopenl3 and KLpasst scores and a higher CLAP score indicate better performance. The results show how the choice of preference dataset affects the model’s ability to generate audio that is both high quality and semantically aligned with the input text.

read the caption

Table 3: Comparison of difference preference dataset used for preference tuning. Metrics include FDopenl3openl3{}_{\text{openl3}}start_FLOATSUBSCRIPT openl3 end_FLOATSUBSCRIPT for Frechet Distance, KLpasstpasst{}_{\text{passt}}start_FLOATSUBSCRIPT passt end_FLOATSUBSCRIPT for KL divergence, and CLAPscorescore{}_{\text{score}}start_FLOATSUBSCRIPT score end_FLOATSUBSCRIPT for alignment.

🔼 This table presents a comparison of different preference datasets used in the preference tuning process of the TANGOFLUX model. The effectiveness of each dataset is evaluated using three metrics: FDopenl3, a measure of Fréchet distance reflecting the overall similarity between the generated audio and a reference dataset; KLpasst, the Kullback-Leibler divergence indicating how closely the generated audio’s probability distribution matches that of the reference data; and CLAP score, an alignment metric assessing how well the generated audio aligns with its corresponding text prompt. The table facilitates the comparison and allows determining which preference dataset leads to the best model performance in terms of audio quality and alignment with textual input.

read the caption

Table 4: Comparison of different preference datasets used for preference tuning. Metrics include FDopenl3openl3{}_{\text{openl3}}start_FLOATSUBSCRIPT openl3 end_FLOATSUBSCRIPT for Fréchet Distance, KLpasstpasst{}_{\text{passt}}start_FLOATSUBSCRIPT passt end_FLOATSUBSCRIPT for KL divergence, and CLAPscorescore{}_{\text{score}}start_FLOATSUBSCRIPT score end_FLOATSUBSCRIPT for alignment.

🔼 Table 5 presents the results of a human evaluation assessing two key aspects of audio generated by different models: Overall audio quality (OVL) and Relevance to the text prompt (REL). To account for potential bias from individual assessors, the data is processed in three ways. First, z-scores normalize the ratings, centering them around the average score for each annotator and scaling by their score spread. Next, the models are ranked based on both the average (mean) and most frequent (mode) ranks. Finally, Elo scores, using a pairwise comparison method, provide a continuous measure of each model’s relative strength compared to others. The table displays these three metrics (z-scores, rankings, and Elo scores) for each model and attribute (OVL and REL). This multi-faceted approach allows for a comprehensive and robust comparison of the models, minimizing the effect of individual assessor bias and providing a nuanced understanding of the relative performance of each.

read the caption

Table 5: Human evaluation results on two attributes: OVL (overall quality) and REL (relevance). We report the z-scores, ranking, and Elo scores to mitigate individual annotator biases and present a relative performance comparison.

🔼 This table presents the results of experiments conducted on the TANGOFLUX model using different classifier-free guidance (CFG) values. The experiments aimed to analyze the impact of CFG on the model’s performance, specifically focusing on the trade-off between audio fidelity (measured by FDopenl3) and semantic alignment (measured by CLAP score). The table shows that varying the CFG scale affects both metrics, indicating an optimal CFG value that balances these two aspects. Lower CFG values generally improve semantic alignment but may compromise audio quality, while higher values can improve audio quality at the cost of reduced semantic accuracy.

read the caption

Table 6: TangoFlux with different classifier free guidance (CFG) values.

🔼 Table 7 provides a detailed list of the 50 prompts used in the human evaluation section of the paper. For each prompt, it indicates whether the prompt contains multiple distinct audio events and whether the temporal order of those events matters for accurate audio generation. This table is crucial for understanding the complexity and diversity of the prompts used to assess the models’ abilities, especially in handling scenarios with multiple events.

read the caption

Table 7: Prompts used in human evaluation and their characteristics.