BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments

🔼 This figure shows the zero-shot performance of different LLMs at various memory footprints. BitStack consistently matches or surpasses the performance of GPTQ and AWQ, especially at extreme compression ratios. The x-axis represents memory usage in MB, and the y-axis represents the average zero-shot performance across six different tasks. The various lines represent different LLMs and compression techniques.

read the caption

(b)

🔼 Figure 1 demonstrates BitStack’s ability to dynamically adjust the size of large language models (LLMs) in environments with varying memory constraints. The left panel (a) shows how BitStack enables fine-grained size control at the megabyte level. The right panel (b) illustrates BitStack’s performance, showing that it achieves comparable or superior results to existing state-of-the-art compression methods such as GPTQ and AWQ, even when operating within the same limited memory footprint.

read the caption

Figure 1: BitStack enables LLMs to dynamically adjust their size in variable memory environments (1(a)) at a megabyte-level, while still matching or surpassing the performance of practical compression methods such as GPTQ (Frantar et al., 2022) and AWQ (Lin et al., 2024) with the same memory footprint(1(b)).

🔼 This figure demonstrates BitStack’s ability to dynamically adjust the size of LLMs in environments with varying memory availability. Panel (a) illustrates how BitStack can adapt to low and high memory scenarios by loading a different number of residual blocks (representing different levels of model compression) at a megabyte-level granularity. This allows the model to seamlessly trade off memory usage and performance as needed.

read the caption

(a)

🔼 The figure shows the zero-shot performance of various LLMs compressed using different methods, including BitStack, GPTQ, and AWQ, across a range of memory footprints. The x-axis represents the memory in MB, and the y-axis represents the average zero-shot performance across six tasks. Different colored lines correspond to different compression methods. The plot highlights the performance of BitStack at various memory levels, demonstrating its ability to match or surpass the performance of other compression techniques at the same memory footprint. The results indicate BitStack’s effectiveness in dynamically adjusting model size and maintaining comparable performance in variable memory environments.

read the caption

(b)

🔼 Figure 2 illustrates BitStack’s dynamic memory management. BitStack uses weight decomposition to create smaller, residual blocks of model weights that can be stored separately on a storage device. When more RAM is available, BitStack loads additional residual blocks from storage to increase model size and performance. Conversely, if available memory decreases, BitStack offloads blocks back to storage. All residual blocks from all layers are stored in a single stack on the storage device, allowing for efficient management. For clarity, the figure omits positional embeddings, normalization, and residual connections.

read the caption

Figure 2: Overview of BitStack. BitStack dynamically loads and offloads weight residual blocks (Figure 3) between RAM and storage devices based on current memory availability. We can load more weight residuals from storage when available memory increases (2(a)), or offload them otherwise (2(b)). The residual blocks for all weights across all layers are universally stored in the same stack on the storage device (grey blocks denote residual blocks for weights in other layers). Note that we omit positional embeddings, normalization layers, and residual connections in the figure for clarity.

🔼 Figure 3 illustrates a residual block, the fundamental unit of data in BitStack’s compression method. Each block is generated through the absolute value decomposition (AVD) of a weight matrix. This process yields two components: a sign matrix containing only +1 or -1 values, and the singular vectors from the singular value decomposition (SVD). The sign matrix is particularly efficient, as its binary nature allows for compact storage using GPU-optimized data types, which reduces memory usage. The figure visually represents these components, showing the original sign matrix and its compressed packed version for storage. The packed sign matrix (denoted by a different symbol) takes up much less memory space than the original sign matrix.

read the caption

Figure 3: Illustration of a residual block in BitStack. A residual block consists of a sign matrix and singular vectors obtained through absolute value decomposition. The sign matrix can be packed into GPU-supported data types to minimize memory usage. denotes the sign matrix while denotes the packed sign matrix.

🔼 This figure demonstrates the performance of BitStack on instruction-tuned Llama 3.1 models of different sizes (8B and 70B) across various tasks in the MT-Bench benchmark. The x-axis represents different memory footprints achieved by loading varying numbers of residual blocks, while the y-axis represents the performance scores. The figure illustrates BitStack’s capability to dynamically adjust the model’s size based on available memory, showcasing its effectiveness across various scales and tasks.

read the caption

(a) Performance with various sizes.

🔼 This figure presents a pairwise comparison of BitStack’s performance against AWQ (Activation-Aware Weight Quantization) across various model sizes (8B and 70B parameters) and different compression ratios. The chart likely displays performance metrics, possibly perplexity scores or accuracy on benchmark tasks, to illustrate how BitStack’s performance compares to AWQ under different memory constraints.

read the caption

(b) Pair-wise comparison with AWQ.

🔼 Figure 4 presents a comprehensive evaluation of BitStack’s performance on instruction-tuned LLMs. Specifically, it uses the MT-Bench benchmark, which assesses performance across various tasks like writing, role-playing, reasoning, and coding. Part (a) shows how the performance of the 8B BitStack model improves as more memory is allocated to it; this demonstrates the fine-grained control BitStack offers. Part (b) provides a direct pairwise comparison of BitStack against AWQ (another compression method) across various compression ratios, for both the 8B and 70B models, highlighting the competitive performance of BitStack.

read the caption

Figure 4: Evaluation results of BitStack Llama 3.1 Instruct 8B/70B models on MT-Bench, assessed by gpt-4o. (4(a)) demonstrates the single-answer grading results across various sizes of the 8B model loaded by BitStack, while (4(b)) illustrates the pairwise comparison results against AWQ at different compression ratios for both the 8B and 70B models.

🔼 Figure 5 presents a comprehensive ablation study on the BitStack model (Llama 3.1 8B). It analyzes the impact of two key components: activation-aware scaling and absolute value decomposition (AVD). The experiments systematically remove one or both of these components, comparing their performance to the full BitStack model. When scaling is omitted, the performance significantly degrades. Similarly, replacing AVD with standard SVD (while adjusting the number of singular values to maintain a comparable residual block size) also causes significant performance drops. This highlights the crucial role of both scaling and AVD in BitStack’s efficiency and accuracy, especially at high compression ratios. The results are shown via perplexity and average zero-shot performance across a range of memory footprints.

read the caption

Figure 5: Perplexity and average zero-shot performance of BitStack Llama 3.1 8B with or without activation-aware scaling and absolute value decomposition(AVD). In the ”w/o scaling” experiments, no scaling is applied as in Eq. 4; in the ”w/o AVD” experiments, vanilla SVD is used instead of AVD as in Eq. 5. For vanilla SVD, we set k′=k+m×n16×(m+n)superscript𝑘′𝑘𝑚𝑛16𝑚𝑛k^{\prime}=k+\frac{m\times n}{16\times(m+n)}italic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_k + divide start_ARG italic_m × italic_n end_ARG start_ARG 16 × ( italic_m + italic_n ) end_ARG(for 𝑾∈ℝm×n𝑾superscriptℝ𝑚𝑛{\bm{W}}\in\mathbb{R}^{m\times n}bold_italic_W ∈ blackboard_R start_POSTSUPERSCRIPT italic_m × italic_n end_POSTSUPERSCRIPT) to ensure the size of each residual block matches that of the main experiments. Solid lines represent average zero-shot performance, while dotted lines represent perplexity scores.

🔼 Figure 6 compares the performance of three different sorting algorithms for residual blocks in the BitStack model, specifically using Llama 3.1 8B. The algorithms are Average, Greedy, and Random. The graph displays both perplexity (dotted lines) and average zero-shot performance (solid lines) across a range of memory footprints. This shows how the choice of sorting algorithm affects the tradeoff between model size and performance.

read the caption

Figure 6: Perplexity and average zero-shot performance of BitStack Llama 3.1 8B with 3 different sorting approaches for residual blocks. Solid lines represent average zero-shot performance, while dotted lines represent perplexity scores.

🔼 This figure demonstrates BitStack’s ability to dynamically adjust its size in environments with varying memory availability. The left panel (a) shows how BitStack adapts its size at a megabyte-level granularity, illustrating the flexibility offered by the approach. Different llama models are shown to have different memory footprints (MB) given their size. The right panel (b) complements this, showing that BitStack maintains or exceeds the performance of other methods (GPTQ, AWQ) at the same memory footprint.

read the caption

(a)

🔼 The figure shows the zero-shot performance of various compressed language models across different memory footprints. BitStack consistently matches or surpasses the performance of GPTQ and AWQ, particularly at extreme compression ratios (low memory). This demonstrates BitStack’s effectiveness in dynamically adjusting model size for optimal performance within variable memory environments. Different colors represent different compression methods.

read the caption

(b)

🔼 Table 2 presents a comprehensive evaluation of the BitStack model’s performance on three different sizes of Llama 2 language models (7B, 13B, and 70B parameters). The evaluation includes two key metrics: perplexity scores (lower is better) on the WikiText2 dataset, a common benchmark for language model performance, and accuracy scores (higher is better) across six zero-shot reasoning tasks. These zero-shot tasks assess the model’s ability to perform reasoning tasks without any explicit training on those specific tasks. The table also shows the memory consumption of the compressed models using BitStack, as well as the corresponding compression ratio (percentage reduction in memory usage compared to the original FP16 model). The results are compared to those obtained using other widely used compression methods like GPTQ and AWQ, allowing for a direct performance comparison. This comprehensive evaluation helps to demonstrate the effectiveness of BitStack in achieving a balance between model size and performance in variable memory environments.

read the caption

Table 2: Evaluation results of Llama 2 7B/13B/70B models. Perplexity scores on WikiText2 test set and accuracy scores on 6 zero-shot reasoning tasks. (↑↑\uparrow↑): higher is better; (↓↓\downarrow↓): lower is better. We denote the overall compression ratio (1−compressed model memoryoriginal model memory1compressed model memoryoriginal model memory1-\frac{\text{compressed model memory}}{\text{original model memory}}1 - divide start_ARG compressed model memory end_ARG start_ARG original model memory end_ARG) after memory consumption.

Model	Memory (MB)	Method	Wiki2 ()	ARC-e ()	ARC-c ()	PIQA ()	HellaS. ()	WinoG. ()	LAMBADA ()	Avg. ()
8B	15316	FP 16	6.13	77.7±0.9	53.3±1.5	80.8±0.9	79.2±0.4	72.7±1.3	76.1±0.6	73.3±0.9
3674(76%)	GPTQw2	1.1e6	25.3±0.9	26.7±1.3	50.6±1.2	26.4±0.4	51.0±1.4	0.0±0.0	30.0±0.9
	AWQw2	1.1e6	25.2±0.9	24.1±1.2	50.7±1.2	26.2±0.4	48.6±1.4	0.0±0.0	29.1±0.9
	BitStack	1.5e3	29.5±0.9	23.9±1.2	53.4±1.2	27.7±0.4	50.6±1.4	0.0±0.0	30.9±0.9
3877(75%)	GPTQw2g128	1.2e5	26.1±0.9	25.9±1.3	50.7±1.2	26.0±0.4	50.0±1.4	0.0±0.0	29.8±0.9
	AWQw2g128	1.7e6	24.8±0.9	24.4±1.3	50.4±1.2	26.4±0.4	50.5±1.4	0.0±0.0	29.4±0.9
	BitStack	96.87	48.5±1.0	25.3±1.3	64.0±1.1	37.1±0.5	56.7±1.4	9.4±0.4	40.2±0.9
4506(71%)	GPTQw3	9.6e4	26.0±0.9	25.7±1.3	50.9±1.2	27.1±0.4	50.3±1.4	0.0±0.0	30.0±0.9
	AWQw3	12.08	61.7±1.0	38.8±1.4	71.4±1.1	68.6±0.5	65.0±1.3	51.9±0.7	59.6±1.0
	BitStack	12.79	69.4±0.9	38.7±1.4	75.6±1.0	63.5±0.5	65.9±1.3	66.6±0.7	63.3±1.0
4709(69%)	GPTQw3g128	8.00	73.1±0.9	46.4±1.5	77.8±1.0	74.5±0.4	71.6±1.3	68.5±0.6	68.7±0.9
	AWQw3g128	8.09	70.7±0.9	44.0±1.5	77.9±1.0	73.4±0.4	70.5±1.3	69.7±0.6	67.7±1.0
	BitStack	11.45	71.6±0.9	42.2±1.4	76.7±1.0	65.8±0.5	67.3±1.3	68.6±0.6	65.4±1.0
5338(65%)	GPTQw4	3.7e4	28.2±0.9	25.3±1.3	51.0±1.2	28.7±0.5	54.6±1.4	0.1±0.0	31.3±0.9
	AWQw4	7.08	75.0±0.9	51.5±1.5	79.5±0.9	77.8±0.4	72.1±1.3	71.1±0.6	71.2±0.9
	BitStack	8.58	74.6±0.9	46.2±1.5	77.5±1.0	72.3±0.4	70.8±1.3	76.0±0.6	69.6±0.9
5541(64%)	GPTQw4g128	1.2e4	31.7±1.0	23.8±1.2	55.1±1.2	29.3±0.5	56.4±1.4	0.7±0.1	32.8±0.9
	AWQw4g128	6.54	76.9±0.9	52.4±1.5	79.9±0.9	78.1±0.4	73.6±1.2	73.6±0.6	72.4±0.9
	BitStack	8.26	75.8±0.9	47.1±1.5	78.7±1.0	73.1±0.4	70.8±1.3	76.3±0.6	70.3±0.9
70B	134570	FP 16	2.85	85.9±0.7	64.3±1.4	84.5±0.8	84.9±0.4	80.7±1.1	79.8±0.6	80.0±0.8
20356(85%)	GPTQw2	3.7e5	24.7±0.9	26.3±1.3	51.5±1.2	26.3±0.4	50.0±1.4	0.0±0.0	29.8±0.9
	AWQw2	8.6e5	25.1±0.9	25.9±1.3	52.3±1.2	26.6±0.4	47.8±1.4	0.0±0.0	29.6±0.9
	BitStack	59.37	46.5±1.0	27.3±1.3	65.2±1.1	39.1±0.5	51.9±1.4	9.2±0.4	39.9±1.0
22531(83%)	GPTQw2g128	4.0e5	25.3±0.9	24.7±1.3	49.3±1.2	26.0±0.4	50.1±1.4	0.0±0.0	29.2±0.9
	AWQw2g128	1.7e6	24.9±0.9	26.4±1.3	51.4±1.2	26.8±0.4	51.8±1.4	0.0±0.0	30.2±0.9
	BitStack	8.86	74.2±0.9	48.4±1.5	78.1±1.0	73.5±0.4	73.6±1.2	71.8±0.6	69.9±0.9
28516(79%)	GPTQw3	NaN	24.6±0.9	25.4±1.3	51.0±1.2	26.2±0.4	50.4±1.4	0.0±0.0	29.6±0.9
	AWQw3	14.04	65.5±1.0	41.2±1.4	73.1±1.0	64.3±0.5	57.4±1.4	46.9±0.7	58.1±1.0
	BitStack	6.88	79.8±0.8	54.8±1.5	80.8±0.9	79.6±0.4	77.0±1.2	75.3±0.6	74.5±0.9
30691(77%)	GPTQw3g128	4.8e5	25.5±0.9	26.5±1.3	51.5±1.2	26.3±0.4	48.8±1.4	0.0±0.0	29.8±0.9
	AWQw3g128	4.59	82.2±0.8	60.6±1.4	82.8±0.9	82.9±0.4	78.4±1.2	76.8±0.6	77.3±0.9
	BitStack	5.69	81.6±0.8	57.8±1.4	82.4±0.9	81.2±0.4	78.5±1.2	79.7±0.6	76.9±0.9
36676(73%)	GPTQw4	NaN	25.2±0.9	25.3±1.3	51.6±1.2	26.3±0.4	50.1±1.4	0.0±0.0	29.8±0.9
	AWQw4	4.16	77.5±0.9	54.4±1.5	81.5±0.9	80.0±0.4	60.5±1.4	67.4±0.7	70.2±0.9
	BitStack	4.88	82.3±0.8	61.1±1.4	83.4±0.9	82.5±0.4	79.9±1.1	80.1±0.6	78.2±0.9
38851(71%)	GPTQw4g128	7.8e5	25.0±0.9	26.3±1.3	49.9±1.2	26.8±0.4	47.4±1.4	0.0±0.0	29.2±0.9
	AWQw4g128	3.23	85.9±0.7	63.5±1.4	84.2±0.9	84.5±0.4	80.1±1.1	78.1±0.6	79.4±0.8
	BitStack	4.80	82.8±0.8	60.2±1.4	82.9±0.9	82.8±0.4	79.6±1.1	80.1±0.6	78.1±0.9

🔼 This table presents a comprehensive evaluation of the BitStack model’s performance on Llama 3 8B and 70B models. It assesses both the perplexity scores (lower is better) on the WikiText2 test set and accuracy scores (higher is better) across six distinct zero-shot reasoning tasks. Crucially, the table highlights the impact of different compression ratios achieved by BitStack, showing how performance varies as the model size is reduced. This allows for a direct comparison of BitStack against other compression techniques on a range of performance metrics, showcasing the efficiency of BitStack at different model sizes.

read the caption

Table 3: Evaluation results of Llama 3 8B/70B models. Perplexity scores on WikiText2 test set and accuracy scores on 6 zero-shot reasoning tasks. (↑↑\uparrow↑): higher is better; (↓↓\downarrow↓): lower is better. We denote the overall compression ratio (1−compressed model memoryoriginal model memory1compressed model memoryoriginal model memory1-\frac{\text{compressed model memory}}{\text{original model memory}}1 - divide start_ARG compressed model memory end_ARG start_ARG original model memory end_ARG) after memory consumption.

🔼 This table presents a qualitative comparison of the BitStack and AWQ model’s performance on a math reasoning task across various compression ratios. It shows the generated responses for each model at different memory footprints (representing various compression levels), highlighting the differences in reasoning capability and correctness as the model size decreases. The goal is to demonstrate BitStack’s ability to maintain reasonable performance even under extreme compression.

read the caption

Table 4: Qualitative results of BitStack Llama-3.1-8B-Instruct at different compression ratios comparing to AWQ.

🔼 This table presents a qualitative comparison of the BitStack and AWQ model compression techniques on the Llama-3.1-70B-Instruct model. The comparison focuses on the quality of generated text responses at various compression ratios, illustrating the performance differences between the two methods under different memory constraints. The table uses examples to showcase how response quality degrades as compression increases, revealing the relative strengths and weaknesses of each approach.

read the caption

Table 5: Qualitative results of BitStack Llama-3.1-70B-Instruct at different compression ratios comparing to AWQ.

🔼 This table shows the size of each residual block in megabytes (MB) for various weight matrices within the BitStack model. The residual blocks are created during the iterative decomposition process, where the original weight matrices are broken down into smaller, manageable units for dynamic loading. The number of singular values retained during singular value decomposition is set to 16 (k=16). The table provides insights into the memory footprint of different weight matrices in BitStack across various model sizes and helps illustrate the fine-grained size control that the model offers.

read the caption

Table 6: Size of residual block in various weight matrices in BitStack (k=16𝑘16k=16italic_k = 16), measures in megabytes(MB).