SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

🔼 This figure illustrates the relationship between computational cost and model size for both Large Language Models (LLMs) and diffusion models. For LLMs, the computation is measured using a context length of 512 tokens and generating 256 output tokens. In contrast, for diffusion models, the computation is calculated for a single step in the generation process. The graph visually represents the rapid increase in computational cost as the model size (measured in billions of parameters) grows. The dashed lines indicate trends, offering insights into the scaling characteristics of these two model types. This helps to visualize the significantly higher computational intensity of diffusion models compared to LLMs of similar parameter counts.

read the caption

Figure 2: Computation vs. parameters for LLMs and diffusion models. LLMs’ computation is measured with 512 context and 256 output tokens, and diffusion models’ computation is for a single step. Dashed lines show trends.

🔼 Figure 3 illustrates the core idea of SVDQuant, a novel 4-bit quantization method for diffusion models. It addresses the challenge of quantizing both activations and weights to 4 bits, which usually leads to significant quality degradation. The figure shows three steps: (a) The initial state, where both activations (X) and weights (W) have outliers, making direct 4-bit quantization difficult. (b) A smoothing technique is applied to shift outliers from activations to weights. This makes the activations easier to quantize, but creates more severe outliers in the weights. (c) SVDQuant decomposes the outlier-rich weights into a low-rank component (L1L2) and a residual. The low-rank component is processed in higher precision (16-bit), while the residual is quantized to 4 bits. This approach significantly reduces quantization errors and maintains image quality.

read the caption

Figure 3: Overview of SVDQuant. (a) Originally, both the activation 𝑿𝑿{\bm{X}}bold_italic_X and weight 𝑾𝑾{\bm{W}}bold_italic_W contain outliers, making 4-bit quantization challenging. (b) We migrate the outliers from the activation to weight, resulting in the updated activation 𝑿^^𝑿\hat{{\bm{X}}}over^ start_ARG bold_italic_X end_ARG and weight 𝑾^^𝑾\hat{{\bm{W}}}over^ start_ARG bold_italic_W end_ARG. While 𝑿^^𝑿\hat{{\bm{X}}}over^ start_ARG bold_italic_X end_ARG becomes easier to quantize, 𝑾^^𝑾\hat{{\bm{W}}}over^ start_ARG bold_italic_W end_ARG now becomes more difficult. (c) SVDQuant further decomposes 𝑾^^𝑾\hat{{\bm{W}}}over^ start_ARG bold_italic_W end_ARG into a low-rank component 𝑳1⁢𝑳2subscript𝑳1subscript𝑳2{\bm{L}}_{1}{\bm{L}}_{2}bold_italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT bold_italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT and a residual 𝑾^−𝑳1⁢𝑳2^𝑾subscript𝑳1subscript𝑳2\hat{{\bm{W}}}-{\bm{L}}_{1}{\bm{L}}_{2}over^ start_ARG bold_italic_W end_ARG - bold_italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT bold_italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT with SVD. Thus, the quantization difficulty is alleviated by the low-rank branch, which runs at 16-bit precision.

🔼 Figure 4 illustrates the effects of SVDQuant’s outlier mitigation process on the weight and activation tensors of the PixArt-Σ model. The figure shows histograms visualizing the distribution of values in the input activation tensor (X), weight tensor (W), and the tensors after applying smoothing (X^, W^) and SVD decomposition (R). Initially, both X and W have significant outliers (represented in red). Smoothing shifts outliers from the activations to the weights, reducing the range of values in X^ but increasing the range and outliers in W^. Finally, the SVD low-rank branch separates the outliers into a low-rank component (L1L2), leaving the residual (R) with a significantly reduced range and no outliers, making it easier to quantize using 4-bit precision.

read the caption

Figure 4: Example value distribution of inputs and weights in PixArt-ΣΣ\Sigmaroman_Σ. 𝝀𝝀{\bm{\lambda}}bold_italic_λ is the smooth factor. Red indicates the outliers. Initially, both the input 𝑿𝑿{\bm{X}}bold_italic_X and weight 𝑾𝑾{\bm{W}}bold_italic_W contain significant outliers. After smoothing, the range of 𝑿^^𝑿\hat{{\bm{X}}}over^ start_ARG bold_italic_X end_ARG is reduced with much fewer outliers, while 𝑾^^𝑾\hat{{\bm{W}}}over^ start_ARG bold_italic_W end_ARG shows more outliers. Once the SVD low-rank branch 𝑳1⁢𝑳2subscript𝑳1subscript𝑳2{\bm{L}}_{1}{\bm{L}}_{2}bold_italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT bold_italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT is subtracted, the residual 𝑹𝑹{\bm{R}}bold_italic_R has a narrower range and is free from outliers.

🔼 Figure 5 illustrates the distribution of the first 64 singular values obtained through Singular Value Decomposition (SVD) for three different weight matrices: the original weight matrix 𝑾 (bold_italic_W), the transformed weight matrix 𝑾̂ (over^ start_ARG bold_italic_W end_ARG), and the residual matrix 𝑹 (bold_italic_R). The plot shows that the transformed weight matrix 𝑾̂ (over^ start_ARG bold_italic_W end_ARG) has a significantly different distribution than the original weight matrix 𝑾 (bold_italic_W). Specifically, the transformed weight matrix has a steeper drop-off in its singular values, where the first 32 values are much larger than the others. The residual matrix 𝑹 (bold_italic_R), on the other hand, exhibits a much more gradual decrease in singular values. This visual representation highlights the effectiveness of the SVD in separating the dominant components from the less significant ones, which forms the basis for the low-rank branch used in the SVDQuant method. The figure directly supports the method’s claim of mitigating outlier effects and reducing the magnitude of values requiring quantization.

read the caption

Figure 5: First 64 singular values of 𝑾𝑾{\bm{W}}bold_italic_W, 𝑾^^𝑾\hat{{\bm{W}}}over^ start_ARG bold_italic_W end_ARG, and 𝑹𝑹{\bm{R}}bold_italic_R. The first 32 singular values of 𝑾^^𝑾\hat{{\bm{W}}}over^ start_ARG bold_italic_W end_ARG exhibit a steep drop, while the remaining values are much more gradual.

🔼 Figure 6 illustrates the performance optimization achieved by Nunchaku, the co-designed inference engine. (a) shows that a naive implementation of the low-rank branch (rank 32) incurs a significant 57% latency overhead due to redundant memory access for both input and output data. Nunchaku addresses this by fusing kernels. (b) details Nunchaku’s kernel fusion strategy: it merges the Down Projection and Quantization kernels because they share the same input data and merges the Up Projection and 4-bit compute kernels as they share the same output data. This fusion significantly reduces data movement and improves efficiency.

read the caption

Figure 6: (a) Naïvely running low-rank branch with rank 32 will introduce 57% latency overhead due to extra read of 16-bit inputs in Down Projection and extra write of 16-bit outputs in Up Projection. Our Nunchaku engine optimizes this overhead with kernel fusion. (b) Down Projection and Quantize kernels use the same input, while Up Projection and 4-Bit Compute kernels share the same output. To reduce data movement overhead, we fuse the first two and the latter two kernels together.

🔼 Figure 7 presents a qualitative comparison of image generation results across different models and quantization methods using the MJHQ dataset. The ‘Image Reward’ metric, calculated across the entire dataset, quantifies the overall quality. For the FLUX.1 model, the 4-bit models (using SVDQuant) outperformed the NF4 W4A16 baseline, showing better text alignment and higher similarity to the 16-bit results. A notable example is the NF4 model’s misinterpretation of the prompt ‘dinosaur style,’ which resulted in an image of a real dinosaur rather than a stylized one. In PixArt-Σ and SDXL-Turbo, the 4-bit models from this work also yielded noticeably better visual quality compared to other state-of-the-art 4-bit methods (ViDiT-Q and MixDQ).

read the caption

Figure 7: Qualitative visual results on MJHQ. Image Reward is calculated over the entire dataset. On FLUX.1 models, our 4-bit models outperform the NF4 W4A16 baselines, demonstrating superior text alignment and closer similarity to the 16-bit models. For instance, NF4 misinterprets “dinosaur style,” generating a real dinosaur. On PixArt-ΣΣ\Sigmaroman_Σ and SDXL-Turbo, our 4-bit results demonstrate noticeably better visual quality than ViDiT-Q’s and MixDQ’s W4A8 results.

🔼 This figure presents a comparison of model size, memory usage, and inference speed between different quantization methods applied to the 12B parameter FLUX.1 diffusion model. It shows that SVDQuant, combined with the Nunchaku inference engine, significantly reduces the model size (by 3.6x compared to the original 16-bit model), memory usage (by 3.5x), and inference time (by 3.0x on desktop GPU and 10.1x on a laptop GPU). The 10.1x speedup on the laptop is attributed to eliminating the need for CPU offloading, a crucial factor in improving the performance of large models on resource-constrained hardware.

read the caption

Figure 8: SVDQuant reduces the model size of the 12B FLUX.1 by 3.6×. Additionally, our engine, Nunchaku, further cuts memory usage of the 16-bit model by 3.5× and delivers 3.0× speedups over the NF4 W4A16 baseline on both the desktop and laptop NVIDIA RTX 4090 GPUs. Remarkably, on laptop 4090, it achieves in total 10.1× speedup by eliminating CPU offloading.

🔼 Figure 9 demonstrates the seamless integration of SVDQuant with off-the-shelf Low-Rank Adapters (LoRAs) without the need for re-quantization. The figure showcases several examples of images generated using the INT4 quantized model with various LoRAs applied. The results show that the INT4 model, even with the LoRAs, maintains the image quality of the original 16-bit FLUX.1-dev model, highlighting the effectiveness of SVDQuant in preserving image quality across different model configurations. Specific prompts used to generate these images can be found in Appendix C.

read the caption

Figure 9: Our INT4 model seamlessly integrates with off-the-shelf LoRAs without requiring requantization. When applying LoRAs, it matches the image quality of the original 16-bit FLUX.1-dev. See Appendix C for the text prompts.

🔼 This ablation study investigates the impact of different quantization methods on the PixArt-Σ image generation model. The experiment uses a low-rank branch with a rank of 64. The performance metric is Image Reward, calculated from 1000 samples of the MJHQ dataset. The results show that SVDQuant significantly outperforms other techniques, such as simple SVD, naïve quantization, smoothing, and LoRC, in terms of image quality. This highlights the effectiveness of SVDQuant’s approach in handling outliers and achieving high-quality results in 4-bit quantization.

read the caption

Figure 10: Ablation study of SVDQuant on PixArt-ΣΣ\Sigmaroman_Σ. The rank of the low-rank branch is 64. Image Reward is measured over 1K samples from MJHQ. Our results significantly outperform the others, achieving the highest image quality by a wide margin.

🔼 This figure shows the trade-off between increasing the rank (r) of the low-rank branch within the SVDQuant model and the resulting impact on image quality, model size, and inference latency. Higher ranks generally lead to better image quality because the low-rank branch can absorb more outliers. However, this improvement comes at the cost of increased model size and latency, making it important to find the optimal balance between image quality and efficiency.

read the caption

Figure 11: Increasing the rank r𝑟ritalic_r of the low-rank branch in SVDQuant can enhance image quality, but it also leads to higher parameter and latency overhead.

🔼 This figure showcases a qualitative comparison of image generation results from the 12B parameter FLUX.1-dev diffusion model using different quantization methods. Specifically, it visually demonstrates the impact of various methods (BF16, NF4 W4A16, SVDQuant INT4, and SVDQuant FP4) on the visual quality of generated images. Each row displays a prompt and the resulting image generated using each method, allowing for direct visual assessment of the quality differences. The prompts and images are selected from the MJHQ dataset, and the goal is to visually demonstrate how well each quantization method preserves the image quality compared to the original, unquantized model.

read the caption

Figure 12: Qualitative visual results of FLUX.1-dev on MJHQ.

🔼 This figure shows a qualitative comparison of image generation results from different models on the MJHQ dataset. Specifically, it compares the quality of images generated by the original 16-bit FLUX.1-schnell model, a weight-only quantized 4-bit version (NF4 W4A16), and the proposed SVDQuant model at 4-bit precision (INT and FP). The prompts used to generate the images are also displayed. The purpose is to visually demonstrate the effectiveness of the proposed method in maintaining image quality despite significant memory and speed improvements.

read the caption

Figure 13: Qualitative visual results of FLUX.1-schnell on MJHQ.

🔼 This figure showcases a qualitative comparison of image generation results using different quantization methods on the PixArt-Σ model. It displays several image prompts and compares the outputs generated using the original FP16 precision model against various 4-bit quantization techniques including ViDiT-Q (INT8 and INT4), and the authors’ SVDQuant method (INT4 and FP4). The goal is to visually demonstrate the effectiveness of SVDQuant in preserving image quality while using significantly reduced precision for weights and activations. The images allow a visual assessment of the fidelity and detail maintained across different quantization methods.

read the caption

Figure 14: Qualitative visual results of PixArt-ΣΣ\Sigmaroman_Σ on MJHQ.

🔼 This figure displays a qualitative comparison of image generation results from the SDXL model using different quantization methods. It showcases several example prompts and their corresponding generated images using the original 16-bit SDXL model and several 4-bit quantized versions, including SVDQuant (ours), TensorRT, and MixDQ. The goal is to visually demonstrate the effectiveness of SVDQuant in maintaining image quality despite aggressive quantization.

read the caption

Figure 15: Qualitative visual results of SDXL on MJHQ.

🔼 This figure displays a qualitative comparison of image generation results from the SDXL-Turbo model (Stable Diffusion XL - Turbo) using different quantization methods on the MJHQ dataset (Midjourney High-Quality dataset). It visually showcases the impact of various 4-bit and 8-bit quantization techniques on the quality of images generated from several prompts. Each row represents a different prompt, and columns show the results for the original FP16 (full precision), the 8-bit quantized versions (MixDQ and SVDQuant), and the 4-bit quantized versions (MixDQ and SVDQuant). The goal is to demonstrate the visual fidelity maintained by the SVDQuant method, even at the aggressive 4-bit quantization level.

read the caption

Figure 16: Qualitative visual results of SDXL-Turbo on MJHQ.

🔼 This figure displays the results of applying five different LoRA (Low-Rank Adaptation) styles to both the original 16-bit FLUX.1-dev model and the INT4 (4-bit integer) quantized version of the model created by SVDQuant. The LoRA styles are Realism, Ghibsky Illustration, Anime, Children’s Sketch, and Yarn Art. The purpose of the figure is to demonstrate that SVDQuant’s 4-bit quantization does not negatively impact image quality when using LoRAs. The visual similarity between the 16-bit and INT4 model outputs across all five LoRA styles supports this conclusion. More detailed text prompts used for image generation are available in Appendix C.

read the caption

Figure 17: Additional LoRA results on FLUX.1-dev. When applying LoRAs, our INT4 model matches the image quality of the original BF16 model. See Appendix C for the detailed used text prompts.