Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment

🔼 This figure visualizes the impact of scaling factor (s) and rank (r) on the performance of SVD-based Low-Rank Adaptation (LoRA). The left panel shows training loss curves, illustrating how different scaling factors affect convergence speed. The right panel displays the gradient norm, demonstrating the relationship between scaling, rank, and gradient magnitude. The figure highlights the trade-off between these factors to improve efficiency and performance of the method. For instance, with low rank (e.g., r=1), the gradient norm is small, while applying proper scaling (s=16) increases it, reducing the performance gap.

read the caption

Figure 2: SVD initialization vs. scaling s𝑠sitalic_s and rank r𝑟ritalic_r

🔼 Figure 3 illustrates the core idea of the GOAT method, comparing it to standard LoRA and full MoE fine-tuning. Single Low-Rank Adaptation shows the basic LoRA method where a weight matrix is reparameterized using low-rank matrices B and A. MoE Fine-tuning depicts a full MoE architecture with multiple experts selected by a router. Subfigure (I) details the GOAT approach: replacing the single (B, A) pair with multiple (Bi, Ai) pairs. These pairs are initialized using different segments of the SVD decomposition of the pre-trained weights (W0) and selected adaptively by the router. Subfigure (II) explains how GOAT aligns optimization with an SVD-structured MoE by individually aligning each expert’s weight and scaling gradients to match full MoE fine-tuning. Wres ensures that the equivalent weight matches the pre-trained weight before optimization begins.

read the caption

Figure 3: Illustration of Our Method. Single Low-Rank Adaptation: LoRA reduces trainable parameters by reparameterizing W𝑊Witalic_W as W=W0+s⁢B⁢A𝑊subscript𝑊0𝑠𝐵𝐴W=W_{0}+sBAitalic_W = italic_W start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_s italic_B italic_A, with B𝐵Bitalic_B and A𝐴Aitalic_A as low-rank matrices. MoE Fine-tuning: Full MoE fine-tuning, where experts W1superscript𝑊1W^{1}italic_W start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT and WEsuperscript𝑊𝐸W^{E}italic_W start_POSTSUPERSCRIPT italic_E end_POSTSUPERSCRIPT are selected by the router in this moment. Subfigure (I): Our method replaces the single pair B,A𝐵𝐴B,Aitalic_B , italic_A with multiple pairs {Bi,Ai}i=1Esuperscriptsubscriptsuperscript𝐵𝑖superscript𝐴𝑖𝑖1𝐸\{B^{i},A^{i}\}_{i=1}^{E}{ italic_B start_POSTSUPERSCRIPT italic_i end_POSTSUPERSCRIPT , italic_A start_POSTSUPERSCRIPT italic_i end_POSTSUPERSCRIPT } start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_E end_POSTSUPERSCRIPT, initialized from different segments of the SVD of W0subscript𝑊0W_{0}italic_W start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and adaptively selected by the router. Subfigure (II): We align optimization with SVD-structured MoE by separately aligning each expert. Wressubscript𝑊resW_{\text{res}}italic_W start_POSTSUBSCRIPT res end_POSTSUBSCRIPT ensures the equivalent weight equals W0subscript𝑊0W_{0}italic_W start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT before optimization, and we scale each expert’s equivalent gradient to closely approximate full MoE fine-tuning.

🔼 This figure displays the training loss curves for various parameter-efficient fine-tuning (PEFT) methods and the full fine-tuning Mixture-of-Experts (MoE) model on the Cars dataset. The x-axis represents the training steps, and the y-axis shows the training loss. The curves demonstrate the convergence speed and final loss achieved by each method. The MoE baselines have their balance loss excluded to allow for a more direct comparison to the single LoRA methods. This visualization allows for a comparison of the training efficiency and effectiveness of different PEFT approaches relative to the full fine-tuning model.

read the caption

Figure 4: Training loss curves of Different LoRA methods and Full Fine-tuning MoE on Cars. The balance loss is excluded in the MoE baselines for a fair comparison with single LoRA baselines.

🔼 This figure shows how the performance of different parameter-efficient fine-tuning (PEFT) methods changes as the rank of the low-rank matrices used in the methods increases. The x-axis represents the rank, ranging from 8 to 128. The y-axis represents the performance, likely measured as accuracy or a similar metric. The figure demonstrates the scalability of the GOAT method, showing how its performance narrows the gap with the full fine-tuning (FT) method as the rank increases. Other methods, such as MoLoRA and HydraLoRA, are also included for comparison.

read the caption

Figure 5: Performance of different methods across ranks.

🔼 This figure demonstrates the impact of the number of experts and the activation ratio on model performance, while keeping the total rank fixed at 32. The x-axis represents different configurations, such as activating 2 out of 8 experts (‘2 in 8’), 4 out of 8, etc., demonstrating how adjusting the number of active experts and their proportion affects overall performance. The y-axis represents the model’s performance, likely measured by accuracy or a similar metric. The graph shows the performance of different methods (likely GOAT and baselines) under various expert configurations, allowing for a comparison of their relative performance under different sparsity levels. The results help to determine an optimal balance between the number of experts and the activation ratio for improved performance and efficiency.

read the caption

Figure 6: Performance vs. number of experts and activation ratio (total rank=32). “2 in 8” means activating 2 out of 8 experts.

🔼 This figure visualizes the distribution of tokens processed by each of the eight experts in a Mixture-of-Experts (MoE) model across various downstream tasks. It shows the proportion of tokens handled by each expert for each task. The goal is to illustrate whether the expert load is balanced across the different tasks or if there’s any skewing towards specific experts for particular task types (e.g., image classification vs. natural language generation). Uniform distribution across experts suggests effective utilization of the MoE architecture and appropriate allocation of tasks among experts.

read the caption

Figure 7: Expert Load Distribution across different tasks. We illustrate the fraction of tokens assigned to each expert {ei}i=18superscriptsubscriptsubscript𝑒𝑖𝑖18\{e_{i}\}_{i=1}^{8}{ italic_e start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 8 end_POSTSUPERSCRIPT

🔼 This table presents the results of evaluating the Llama-2-7B language model on three distinct tasks: MT-Bench (a multi-turn dialogue benchmark), GSM8K (a grade-school math problem solving benchmark), and HumanEval (a code generation benchmark). The table compares the performance of the Llama-2-7B model on these tasks, likely showing various metrics such as accuracy, precision, or F1-score. The purpose is to demonstrate the model’s capabilities in different areas of natural language processing.

read the caption

Table 2: We evaluate Llama-2-7B on MT-Bench, GSM8K, and HumanEval for dialogue, math, and coding.

🔼 This table presents a performance comparison of different methods for commonsense reasoning on the LLaMA2 7B model. It evaluates eight commonsense reasoning datasets and compares the performance of the GOAT model with several baseline methods, including single LoRA models and LORA MoE models. Results are shown as percentages, and the symbol † indicates results taken from other published works. The table highlights the superior performance of GOAT compared to other methods, demonstrating improved accuracy across various commonsense reasoning tasks.

read the caption

Table 3: Performance comparison of LLaMA2 7B with different methods on eight commonsense reasoning datasets. The symbol ††\dagger† indicates that the results are taken from (Wang et al., 2024a; Zhong et al., 2024; Si et al., 2024).

🔼 This table presents a comparison of the performance of different fine-tuning methods on seven tasks from the GLUE benchmark, using the RoBERTa-large model. The methods compared include full fine-tuning (Full FT), full fine-tuning with a Mixture of Experts (MoE) architecture (Full FT MoE), and several parameter-efficient fine-tuning (PEFT) methods. For each method, the table shows the percentage of parameters used compared to the full fine-tuning model and the accuracy achieved on each of the seven GLUE tasks. The total rank of low-rank adapters was set to 32 for all PEFT methods. This allows for a comparison of performance relative to the parameter count for each method.

read the caption

Table 4: Performance comparison of RoBERTa-large with different methods on 7 GLUE tasks. Total rank is set to 32.

🔼 This ablation study analyzes the impact of different components of the GOAT model on its performance. Specifically, it compares the performance of GOAT using various initialization strategies for the low-rank adapters: the proposed adaptive prior selection (‘O’), using only the principal singular values (‘P’), only the minor singular values (‘M’), and random initialization (‘R’). It also investigates the effect of applying MoE scaling (‘MS’) on top of these initialization methods, both with and without the full MoE architecture. The results help determine the contribution of each component to GOAT’s improved efficiency and accuracy.

read the caption

Table 5: Ablation study of GOAT. “MoE” denotes using the MoE architecture instead of a single LoRA. “MS” refers to using MoE scaling. “O”, “P”, “M”, and “R” represent initializations from segments that selected by ours, with the principal singular value, with the minor singular value, and are randomly selected, respectively.

🔼 This table presents a comparison of the performance achieved by three different models (MoLoRA, HydraLoRA, and GOAT) across various learning rates. It shows how sensitive each model’s performance is to changes in learning rate and illustrates the relative robustness of GOAT compared to the other two models.

read the caption

Table 6: Performance comparison of different learning rates.

🔼 This table compares the performance of LoRA-based Mixture-of-Experts (MoE) models against the full fine-tuned MoE model. It shows a comparison across three key metrics: memory usage (in GB), training time (hours and minutes), and the performance on the GSM8K benchmark. All experiments were conducted using a single NVIDIA A100 GPU with consistent batch sizes to ensure fair comparison.

read the caption

Table 7: Comparison of LoRA-MoE and Full FT MoE in memory cost, training time, and GSM8K performance. Memory cost was measured and training time was recorded on the MetaMath dataset using one A100 GPU with identical batch sizes.

🔼 This table presents a comparison of the performance of the original GOAT method and its extended version, GOAT+, which is adapted for scenarios with proper scaling. The comparison is made across four different benchmark categories (NLG, NLU, IC, and CR), showing average scores for each method. This demonstrates the robustness and generalizability of the GOAT approach, even when the scaling factor is pre-adjusted.

read the caption

Table 8: Performance comparison of our method extended to properly scaled scenarios.

🔼 This table lists the hyperparameters used for the commonsense reasoning task when training the GOAT model. It includes details such as batch size, rank, alpha, optimizer, warmup steps, dropout rate, learning rate, and the number of epochs. These settings are crucial for optimizing the model’s performance on this specific task.

read the caption

Table 9: Hyperparameters of the commonsense reasoning task for GOAT.

🔼 This table details the hyperparameters used for training the GOAT model on image classification tasks. It shows the values used for different parameters including batch size, rank, alpha, optimizer, warmup steps, dropout, learning rate, and epochs. The specific values are listed for each of the seven datasets used in the image classification experiments: Cars, DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN. This information helps readers understand the experimental setup and how those parameters were chosen for the experiments.

read the caption

Table 10: Hyperparameters of the image classification task for GOAT.

🔼 This table details the hyperparameters used for training the GOAT model on seven natural language understanding tasks from the GLUE benchmark. It lists the batch size, rank, alpha value, optimizer, warmup steps, dropout rate, learning rate, and number of epochs used for each task. This information is crucial for reproducibility and understanding the training process of the GOAT model on these specific datasets.

read the caption

Table 11: Hyperparameters of the natural language understanding tasks for GOAT.

🔼 This table presents the hyperparameters used for training the GOAT model on natural language generation tasks. It details the specific settings for batch size, rank, alpha (a scaling factor), optimizer (AdamW), warmup steps, dropout rate, learning rate, and number of epochs. These parameters were crucial in optimizing the model’s performance on these specific tasks.

read the caption

Table 12: Hyperparameters of the natural language generation task for GOAT.