BitNet a4.8: 4-bit Activations for 1-bit LLMs

↗ arXiv ↗ Hugging Face ↗ Papers with Code

TL;DR

#

Key Takeaways

#

Why does it matter?

#

This paper is important because it significantly advances the efficiency of large language models (LLMs). By achieving comparable performance to existing 1-bit LLMs while using 4-bit activations, it offers a substantial improvement in inference speed. This is crucial for deploying LLMs in resource-constrained environments, making them more accessible for wider applications. The research opens up new avenues for exploring hybrid quantization and sparsification techniques in model optimization, contributing to future developments in LLM efficiency.

Visual Insights

#

🔼 BitNet a4.8 uses a hybrid approach to quantization, combining both weight and activation quantization. The weights are ternary (1.58 bits), as in its predecessor BitNet b1.58. However, activations are handled differently. Inputs to the attention and feed-forward network (FFN) layers utilize 4-bit quantization. To manage potential errors caused by outlier values in certain Transformer sub-layers, intermediate states undergo a sparsification process followed by 8-bit quantization. This combination aims to balance quantization efficiency with accuracy.

read the caption

Figure 1: The overview of BitNet a4.8 with both weight and activation quantization. All the parameters are ternery (i.e., 1.58-bit as in BitNet b1.58 [12]). We use a hybrid quantization and sparsification strategy to deal with outlier activations in certain Transformer sub-layers.

🔼 This table presents a comparison of the performance of three different language models: BitNet a4.8, BitNet b1.58, and LLaMA, across various sizes (700M, 1.3B, 3B, and 7B parameters). For each model size, it shows the perplexity (PPL) score on a held-out dataset and zero-shot accuracy scores on several downstream tasks: ARC-Challenge (ARCc), ARC-Easy (ARCe), HellaSwag (HS), PIQA (PQ), Winogrande (WGe). The average of these downstream task scores is also provided. The table highlights the comparable performance of BitNet a4.8 to BitNet b1.58, despite BitNet a4.8 using 4-bit activations, suggesting improved efficiency.

read the caption

Table 1: Perplexity and results of BitNet a4.8, BitNet b1.58 and LLaMA LLM on the end tasks. The standard variance of error for average scores is 1.06%.

In-depth insights

#

4-bit Activation

#

The concept of “4-bit activation” in the context of large language models (LLMs) represents a significant advancement in efficient model design. Reducing the bit-width of activations from the typical 8-bit or even higher precision to 4-bit drastically decreases computational costs and memory usage. This is especially crucial for inference, where latency and resource consumption are primary concerns. However, simply truncating the precision of activations leads to significant information loss and performance degradation. The research likely explores sophisticated quantization techniques to minimize this loss, potentially involving methods like hybrid quantization and sparsification. These techniques cleverly combine different quantization strategies across layers or parts of the model, for example, using 4-bit quantization only on certain input layers and switching to lower or higher bit-widths in other layers. Sparsification, which involves setting a significant number of activation values to zero, reduces computational complexity further. The effectiveness of such hybrid approaches lies in cleverly addressing the challenges posed by outlier activations, which disproportionately affect model accuracy when aggressively quantized. The paper likely benchmarks the performance of these methods against full-precision models and demonstrates a satisfactory accuracy/efficiency trade-off.

Hybrid Quantization

#

Hybrid quantization, as discussed in the context of the research paper, presents a sophisticated approach to address the challenges of low-bit quantization in large language models (LLMs). It strategically combines different quantization techniques to leverage their respective strengths while mitigating their weaknesses. The core idea is to selectively apply various bit-widths to different parts of the model based on the characteristics of the data or the layer’s function. This adaptive approach is crucial because activations in LLMs often exhibit a long-tailed distribution, with outlier values causing significant quantization errors. By employing a hybrid scheme, the technique can effectively quantize typical activations with lower bit-widths (e.g., 4-bit) while handling outliers using a higher bit-width (e.g., 8-bit) or sparsification to improve accuracy. The results demonstrate the effectiveness of this method in maintaining model performance and significantly reducing computational costs, offering a powerful trade-off between efficiency and accuracy. Hybrid quantization represents a step forward in optimizing LLMs for real-world deployment, allowing for faster inference without sacrificing performance. The technique’s adaptability makes it particularly well-suited for diverse model architectures and datasets.

Sparsification Strategy

#

The effectiveness of a sparsification strategy hinges on its ability to selectively remove less-important activations without significantly impacting model performance. A successful strategy must carefully consider the distribution of activations. Simply removing activations based on magnitude alone might prove insufficient, as it could inadvertently discard crucial information. The paper’s hybrid approach is intriguing, combining quantization with sparsification to address this challenge. By targeting outlier channels, which while sparse in number exert disproportionate influence, the strategy aims to minimize quantization errors. The choice of 8-bit quantization for intermediate states, while sparsifying, suggests a balance between accuracy preservation and computational savings. The strategy’s efficacy is further enhanced by its integration within the training process, using a two-stage recipe to adapt the model to the lower-bit activations, ensuring a seamless transition.

Training Efficiency

#

Training efficiency is a crucial aspect of large language model (LLM) development, impacting both time and resource consumption. The paper’s focus on BitNet a4.8 highlights strategies to enhance this efficiency. Continue-training from a pre-trained model (BitNet b1.58) is employed, reducing the need for extensive training from scratch. A two-stage training recipe further streamlines the process by first training with 8-bit activations and then transitioning to the target 4-bit activations. This approach minimizes training time and computational costs. The use of straight-through estimator (STE) for gradient approximation simplifies the training process by bypassing non-differentiable functions. Finally, mixed-precision training, which combines different precision levels during the training, allows efficient utilization of hardware resources. This combination of techniques demonstrates that significant gains in training efficiency can be achieved without sacrificing model performance.

LLM Deployment

#

Efficient Large Language Model (LLM) deployment is crucial for realizing their full potential. Reducing inference costs is paramount, and techniques like quantization (reducing the numerical precision of model weights and activations) and sparsification (reducing the number of non-zero parameters) are vital. The paper highlights BitNet a4.8’s efficiency gains through 4-bit activations and a hybrid quantization/sparsification strategy, targeting outlier channels to mitigate quantization errors. Faster inference is achieved using 4-bit kernels. Furthermore, the model’s reduced parameter count (55% activation) and support for 3-bit KV cache significantly lower memory demands and improve deployment efficiency in large-scale scenarios. These optimizations are critical for deploying LLMs on resource-constrained devices or within cost-effective cloud infrastructures.

More visual insights

#

More on figures

🔼 Figure 2 presents the distribution patterns of inputs for different projections within a 7B parameter BitNet b1.58 model, using a subset of the C4 validation dataset. The visualizations reveal that some layers exhibit Gaussian-like distributions, leading to the application of 4-bit activation quantization. Other layers, however, show sharp distributions, prompting the use of Q-Sparse (a technique described in reference [18] of the paper) for sparsification of the activations. This figure highlights the model’s adaptive quantization strategy based on the input distribution characteristics.

read the caption

Figure 2: The distribution of the inputs to each projection. The visualization is conducted with a 7B BitNet b1.58 model on a subset of the valid set of C4. For the layers that exhibit Gaussian-like distributions, we employ 4-bit activation quantization. For the layers which distributions are sharp, we adopt Q-Sparse [18] to perform sparsification on the activations.

🔼 This figure compares the distribution of inputs to the output projection of the attention mechanism in a 7B parameter BitNet b1.58 model under different quantization and sparsification strategies. It visualizes the impact of these techniques on the distribution of activation values. The three subplots show how using INT8 (8-bit integer) quantization, INT4 (4-bit integer) quantization, and a combined INT8 quantization with TopK 50% sparsification affects the activation value distribution. This helps illustrate the effectiveness of the hybrid quantization and sparsification approach in managing outliers and maintaining performance. The data is from a subset of the valid set of C4.

read the caption

Figure 3: The distribution of the inputs to the output projection of attention with different quantization and sparsification. The visualization is conducted with a 7B BitNet b1.58 model on a subset of the valid set of C4.

🔼 This figure presents an ablation study comparing the performance of different quantization strategies on a language model. It shows the training loss curves for a model trained with full 4-bit integer (INT4) quantization, full 4-bit floating-point (FP4) quantization, and the hybrid quantization and sparsification approach (A4.8) proposed in the paper. The graph visualizes how the training loss changes over the number of training tokens used, allowing comparison of the different approaches.

read the caption

Figure 4: Ablation study on the hybrid quantization and sparsification.

🔼 This ablation study investigates the impact of various quantization methods (INT4, FP4, INT8) and activation functions (Swish, ReLU2) on the inputs to the feed-forward network’s (FFN) down projection. The figure displays the training loss curves for different configurations, allowing for a comparison of their performance in terms of training perplexity. It helps determine the optimal combination of quantization and activation function for this specific layer of the model.

read the caption

Figure 5: Ablation study on different quantization or activation function for the inputs to down projection of FFN.

🔼 This figure presents an ablation study comparing different 4-bit quantization methods for the inputs of the attention and feed-forward network (FFN) layers. It shows the training loss curves for various quantization techniques, including floating-point quantization with different exponent and mantissa bit configurations (E1M2 and E2M1 formats using MinMax quantizer), and integer quantization using absmax and absmean quantizers. The goal is to determine which 4-bit quantization approach yields the best training performance for these critical layers in the model.

read the caption

Figure 6: Ablations on 4-bit quantizers for the inputs to attention and FFN.

Full paper

#