XMusic: Towards a Generalized and Controllable Symbolic Music Generation Framework

🔼 Figure 2 illustrates the XMusic framework, a system for generating high-quality symbolic music from various prompts. Panel (a) shows the diverse input types: images, videos, text, tags, and humming. The XProjector component (b) processes these prompts, translating them into symbolic music elements (emotions, genres, rhythms, and notes) within a common ‘Projection Space’. Next, the XComposer component’s Generator (c) uses these elements to generate the actual music, employing a Transformer Decoder to create sequential musical representations. Finally, the Selector (d) assesses the generated music’s quality via a multi-task learning process incorporating emotion recognition, genre recognition, and overall quality evaluation.

read the caption

Figure 2: Illustration of the proposed XMusic, which supports flexible (a) X-Prompts to guide the generation of high-quality symbolic music. The XProjector analyzes these prompts, mapping them to symbolic music elements within the (b) Projection Space. Subsequently, the (c) Generator of XComposer transforms these symbolic music elements into token sequences based on our enhanced representation. It employs a Transformer Decoder as the generative model to predict successive events iteratively, thereby creating complete musical compositions. Finally, the (d) Selector of XComposer utilizes a Transformer Encoder to encode the complete token sequences and employs a multi-task learning scheme to evaluate the quality of the generated music.

🔼 Figure 3 illustrates the differences between our proposed symbolic music representation and the Compound Word (CP) representation introduced in a previous work [10]. The core enhancement in our representation is the inclusion of new token types (shown in dotted boxes) which provide explicit control over musical elements, namely: genre, emotion, and instrument type. These additional tokens improve the model’s ability to generate diverse and controlled symbolic music. The addition of the ‘Tag’ token, encapsulating ‘Emotion’ and ‘Genre’, allows for high-level semantic control over style and emotional expression. The ‘Instrument’ token, further divided into ‘Program’, facilitates the creation of multi-track compositions. The improvements to the ‘Rhythm’ token allows more precise control over rhythmic elements, namely ‘density’ and ‘strength’.

read the caption

Figure 3: Comparison between our representation and Compound Word (CP)  [10] representation. The dotted boxes represent our new tokens in comparison with those of the CP representation.

🔼 Figure 4 presents a visual summary of the XMIDI dataset’s key statistical properties. It consists of three subfigures: (a) illustrates the distribution of the 11 emotion classes, showing the relative frequency of each emotion in the dataset. (b) shows the distribution of the 6 genre classes, similarly displaying their proportions. (c) illustrates the distribution of music lengths across the dataset, indicating the frequency of different song durations. This figure provides a concise overview of the dataset’s characteristics in terms of emotional variety, musical genres and song lengths.

read the caption

Figure 4: Data statistics of our XMIDI dataset.

🔼 This table details the specific implementation choices made in the design of the symbolic music representation used in the XMusic model. It compares the vocabulary size, embedding size, and token types in the proposed representation with those used in the Compound Word (CP) model from a previous work. This allows the reader to better understand the differences in the approach and scale of these two models.

read the caption

TABLE II: Comparison of the implementation details between the CP and our proposed representation.

🔼 This table compares the characteristics of several existing datasets of emotion-labeled MIDI files with the new XMIDI dataset introduced in this paper. It shows the number of emotion classes, genre types (or whether genre information is included at all), and the total number of MIDI files included in each dataset. This comparison highlights the scale and comprehensiveness of the XMIDI dataset relative to prior work, particularly its significantly larger size and more detailed annotation (e.g., 11 emotion classes vs. fewer in others).

read the caption

TABLE III: Comparison between existing emotion-labeled MIDI datasets and the proposed XMIDI dataset.

🔼 This table presents an objective comparison of XMusic with state-of-the-art symbolic music generation methods across three key metrics: Pitch Class Histogram Entropy (PCE), Grooving Pattern Similarity (GS), and Empty Beat Rate (EBR). Lower PCE and EBR scores indicate better tonality and rhythm, while a higher GS score suggests stronger rhythmic structure. The table is divided into two sections: unconditioned and video-conditioned methods, showcasing XMusic’s superior performance in both scenarios.

read the caption

TABLE IV: Objective comparison with state-of-the-art symbolic music generation methods.

🔼 This table presents a subjective evaluation comparing the performance of different symbolic music generation methods. Three state-of-the-art methods and the proposed XMusic framework are compared across four categories of music generation (unconditioned, video-conditioned, text-conditioned, and image-conditioned), with human participants rating the generated music samples based on metrics such as richness, correctness, structuredness, emotion matching, and rhythm matching. The average ranking for each method within each category provides a comprehensive comparison of their overall quality and strengths across different generation paradigms.

read the caption

TABLE V: Subjective comparison with state-of-the-art symbolic music generation methods.

🔼 This table presents a subjective comparison of XMusic with the state-of-the-art emotion-conditioned symbolic music generation method, EMOPIA. It shows the percentage of correctly classified music pieces based on four emotion categories: Positive Valence (PV), Negative Valence (NV), Positive Arousal (PA), and Negative Arousal (NA). These categories are derived from a combination of EMOPIA’s emotion quadrants and XMusic’s more granular emotion labels, allowing for a more meaningful comparison despite the difference in granularity.

read the caption

TABLE VI: Subjective comparison with state-of-the-art emotion-conditioned symbolic music generation method.

🔼 This table presents the results of an ablation study on the XMusic model. It systematically removes or modifies different components of the model to assess their individual contributions to overall performance. Specifically, it examines the impact of the Selector, emotion control level (music-level vs. bar-level), music representation (comparing different token schemes), and data sampling strategies (undersampling and oversampling) on the subjective and objective evaluation metrics. The subjective metrics used include Richness, Correctness, Structuredness, Emotion-Matching, Rhythm-Matching, and Overall Rank, all based on human evaluations. The table provides a detailed quantitative analysis of how each modification affects the performance, allowing for a better understanding of the key components within the XMusic framework.

read the caption

TABLE VII: Ablation analysis.

🔼 Table VIII presents a comparison of the performance of XMusic against state-of-the-art symbolic music generation methods on publicly available datasets. Specifically, it shows a subjective evaluation of the quality of music generated by XMusic compared to CP [10] and EMOPIA [11] across different metrics such as Richness, Correctness, and Structuredness. The evaluation was conducted on two datasets: AILabs1k7 and EMOPIA.

read the caption

TABLE VIII: Comparison with state-of-the-art symbolic music generation methods on public datasets.

🔼 This table presents the results of an ablation study evaluating the impact of the proposed Selector on the overall performance of the XMusic model. It compares the objective metrics (PCE, GS, EBR) achieved by the XMusic model with and without the Selector. This demonstrates the effectiveness of the Selector in enhancing the quality of generated symbolic music.

read the caption

TABLE IX: Objective ablation study on the proposed Selector.

🔼 This table presents an ablation study evaluating the impact of the multi-task learning scheme on the Selector component of the XMusic model. It shows how the accuracy of the quality assessment task changes when different combinations of sub-tasks (quality, emotion, and genre recognition) are included in the training process. This demonstrates the contribution of each sub-task to the overall performance of the Selector in identifying high-quality music.

read the caption

TABLE X: Ablation study on the multi-task learning scheme of the Selector.

🔼 This table presents the results of an ablation study investigating the impact of different weighting factors (λ1 and λ2) on the accuracy of image emotion recognition. The study varied the weights assigned to two different models used for emotion classification (ResNet and CLIP) in the XProjector component of the XMusic system. The table shows how changes in these weights affect the overall accuracy of image emotion recognition, providing insights into the relative contributions of each model.

read the caption

TABLE XI: Ablation analysis on the weighting factors λ1subscript𝜆1\lambda_{1}italic_λ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and λ2subscript𝜆2\lambda_{2}italic_λ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT of image emotion recognition task.