Matryoshka Quantization

2502.06786

Pranav Nair et el.

🤗 2025-02-11

TL;DR
#

Training large language models (LLMs) is computationally expensive, and deploying them on resource-constrained devices requires model compression techniques like quantization. However, traditional quantization methods often lead to a significant trade-off between model accuracy and precision, especially at lower precision levels (e.g. int2). Practitioners typically use multiple models to balance accuracy and latency, increasing the complexity of deployment.

This research introduces Matryoshka Quantization (MatQuant), a novel method that addresses these limitations. MatQuant leverages the nested structure of integer data types to jointly optimize model weights across multiple precision levels (int8, int4, int2). This allows for extracting multiple accurate lower-precision models from a single trained model. Results show significant improvements in the accuracy of int2 models, outperforming standard int2 quantization techniques by up to 10%, while maintaining comparable accuracy at higher precisions. Additionally, MatQuant seamlessly extracts accurate models for intermediate precision levels without requiring separate training.

Key Takeaways
#

Why does it matter?
#

This paper is important because it introduces a novel quantization technique that significantly improves the accuracy of low-precision models, which is crucial for deploying large language models efficiently on resource-constrained devices. It addresses the challenge of needing multiple models with different quantization levels by enabling the training and maintenance of a single model. This also opens avenues for future research, particularly in exploring multi-scale training techniques and their application to other areas of deep learning.

Visual Insights
#

🔼 This figure shows a multi-scale quantization training technique leveraging the nested structure of integer data types. Specifically, it illustrates how smaller bit-width integers (int4, int2) are nested within larger ones (int8). This nested structure is exploited to train a single model that can be efficiently served at different precision levels, addressing the challenge of needing multiple quantized models for different accuracy/latency trade-offs.
read the caption
(a)

Data type	Method	Gemma-2 2B		Gemma-2 9B		Mistral 7B
	OmniQuant	Task Avg.	log pplx.	Task Avg.	log pplx.	Task Avg.	log pplx.
bfloat16		$68.21$	$2.551$	$74.38$	$2.418$	$73.99$	$2.110$
int8	Baseline	$68.25$	$2.552$	$74.59$	$2.418$	$73.77$	$2.110$
int8	${\rm MatQuant}$	$67.85$	$2.580$	$74.33$	$2.446$	$73.46$	$2.132$
int4	Sliced int8	$62.98$	$2.794$	$72.19$	$2.546$	$46.59$	$4.139$
	Baseline	$67.03$	$2.598$	$74.33$	$2.451$	$73.62$	$2.136$
	${\rm MatQuant}$	$66.54$	$2.617$	$74.26$	$2.470$	$73.13$	$2.155$
int2	Sliced int8	$37.68$	$17.993$	$35.75$	$14.892$	$36.25$	$10.831$
	Baseline	$51.33$	$3.835$	$60.24$	$3.292$	$59.74$	$3.931$
	${\rm MatQuant}$	$\bf 55.70$	$\bf 3.355$	$\bf 68.25$	$\bf 2.823$	$\bf 65.99$	$\bf 2.569$
int6	Sliced int8	$67.66$	$2.565$	$74.61$	$2.424$	$73.50$	$2.122$
	Baseline	$68.06$	$2.554$	$74.23$	$2.420$	$74.10$	$2.112$
	${\rm MatQuant}$	$68.01$	$2.582$	$74.50$	$2.446$	$73.59$	$2.139$
int3	Sliced int8	$42.00$	$5.781$	$55.76$	$3.830$	$34.60$	$8.539$
	Baseline	$64.37$	$2.727$	$73.23$	$2.549$	$71.68$	$2.211$
	${\rm MatQuant}$	$63.24$	$2.757$	$73.25$	$2.535$	$71.55$	$2.228$

🔼 This table presents a comparison of the performance of Matryoshka Quantization (MatQuant) against baseline methods for quantizing the Gemma-2 (2B and 9B) and Mistral-7B LLMs. MatQuant is evaluated at various bit-widths (int8, int4, int2, int6, int3). The results show that MatQuant achieves comparable performance to the baselines for int8 and int4, but substantially improves accuracy for int2. Importantly, MatQuant’s multi-scale training allows for the extraction of int3 and int6 models with accuracy comparable to independently trained baselines, demonstrating its efficiency.
read the caption
Table 1: MatQuantMatQuant{\rm MatQuant}roman_MatQuant with OmniQuant across Gemma-2 2B, 9B and Mistral 7B models. MatQuantMatQuant{\rm MatQuant}roman_MatQuant performs on par with the baseline for int4 and int8 while significantly outperforming it for int2. Even the int3, int6 models obtained for free through interpolation from MatQuantMatQuant{\rm MatQuant}roman_MatQuant perform comparably to the explicitly trained baselines. Task Avg. is average accuracy on the evaluation tasks (↑↑\uparrow↑) while log pplx (perplexity) is computed on C4 validation set (↓↓\downarrow↓).

In-depth insights
#

MatQuant: Multi-Scale
#

MatQuant, a multi-scale quantization technique, presents a novel approach to model quantization by leveraging the nested structure of integer data types. Instead of training separate models for different precision levels (e.g., int8, int4, int2), MatQuant trains a single model that can be efficiently served at various precisions. This is achieved by jointly optimizing the model’s weights across multiple precisions. The inherent Matryoshka structure allows smaller bit-width integers to be nested within larger ones, enabling seamless extraction of lower-precision models from the trained model. This multi-scale approach results in significant improvements in accuracy, especially for low-precision quantization (int2), outperforming traditional methods by up to 10%. Further, MatQuant demonstrates adaptability through interpolation and Mix’n’Match strategies, offering flexibility in balancing accuracy and resource consumption. This work is of high significance as it addresses the trade-off between model accuracy and efficiency inherent in model quantization, offering a more versatile and computationally efficient approach to deploying large language models.

Int2 Accuracy Boost
#

The research paper’s focus on achieving an ‘Int2 Accuracy Boost’ is a significant contribution to model quantization. Standard int2 quantization methods often suffer from substantial accuracy loss, making them impractical for many applications. The paper’s novel approach, likely involving a multi-scale training technique and/or innovative regularization, directly addresses this limitation. The reported 10% improvement over standard int2 methods is remarkable, showcasing a clear advancement in the field. This finding suggests that the proposed technique effectively mitigates the information loss inherent in lower-bit quantization. The method’s general-purpose nature, applicable across various models, further amplifies its significance, implying a potentially wide-ranging impact on resource-constrained deployments of large language models and other deep learning architectures. The success hinges on effectively harnessing the nested structure of integer data types to enable seamless extraction of multiple precision models. This breakthrough has important implications for model deployment, enabling a trade-off between accuracy and inference speed tailored to specific hardware constraints.

Mix’n’Match Quantization
#

Mix’n’Match quantization, as a concept, offers a powerful approach to optimize model efficiency. By allowing different layers of a neural network to utilize varying quantization precisions, it dynamically balances accuracy and resource consumption. Instead of applying a uniform quantization scheme across the entire model, this technique strategically chooses the most appropriate precision for each layer based on its sensitivity to quantization. Layers requiring high accuracy retain higher precision (e.g., int8), while less critical layers can leverage lower precisions (e.g., int4 or int2) to significantly reduce computational costs and memory footprint. This adaptability makes Mix’n’Match particularly useful for deployment on hardware with diverse capabilities, allowing for customized optimization based on the specific resource constraints of the target platform. The ability to seamlessly combine various precisions is a key advantage, offering a more granular control over the model’s efficiency compared to uniform quantization. Furthermore, research into optimal strategies for layer-wise precision selection is an important area for future exploration, considering factors like layer importance, activation patterns, and computational demands. The flexibility and potential performance gains of Mix’n’Match quantization make it a promising area of research for improving the efficiency and deployment of deep learning models across various hardware platforms.

Interpolative Behavior
#

The concept of “Interpolative Behavior” in the context of Matryoshka Quantization is a significant finding. It demonstrates that a model trained to optimize across multiple precision levels (e.g., int8, int4, int2) can be effectively used at intermediate precisions. Simply slicing the higher-precision model’s most significant bits to extract lower-precision versions yields surprisingly accurate results, comparable to models specifically trained at those intermediate levels. This is a crucial efficiency gain as it eliminates the need to train separate models for each desired precision. The nested structure of integer data types is leveraged here, showing that the knowledge acquired during multi-scale training is transferable and robust enough to provide good performance even when not explicitly trained for those precisions. This interpolative property significantly reduces the computational cost and storage requirements associated with model quantization. This characteristic of Matryoshka Quantization allows for flexible and cost-effective deployment, making it particularly attractive for resource-constrained environments where serving multiple models is impractical.

Future: FP Extension
#

Extending Matryoshka Quantization (MatQuant) to floating-point (FP) numbers presents a significant challenge. Unlike integers, where slicing MSBs directly yields lower precision representations, FP numbers have an exponent and mantissa. Slicing the exponent results in exponentially increasing bucket sizes, disrupting the nested structure crucial to MatQuant’s multi-scale optimization. This makes achieving a seamless accuracy-cost tradeoff difficult as observed in integer quantization. Addressing this would require innovative techniques that handle the non-linear relationship between exponent and mantissa values during quantization. This might involve new loss functions or quantization schemes that preserve the accuracy of the exponent’s contribution. Successful FP extension of MatQuant could unlock its benefits for a wider range of hardware and model architectures, expanding its applicability beyond integer-based systems and making low-precision inference even more efficient and versatile.

More visual insights
#

More on tables

Data type	Method	Gemma-2 2B		Gemma-2 9B		Mistral 7B
	QAT	Task Avg.	log pplx.	Task Avg.	log pplx.	Task Avg.	log pplx.
bfloat16		$68.21$	$2.551$	$74.38$	$2.418$	$73.99$	$2.110$
int8	Baseline	$67.82$	$2.458$	$74.17$	$2.29$	$73.48$	$2.084$
int8	${\rm MatQuant}$	$67.68$	$2.471$	$74.77$	$2.301$	$72.41$	$2.085$
int4	Sliced int8	$67.20$	$2.458$	$73.25$	$2.338$	$71.83$	$2.164$
	Baseline	$67.03$	$2.512$	$73.26$	$2.324$	$72.13$	$2.105$
	${\rm MatQuant}$	$67.05$	$2.521$	$73.71$	$2.332$	$71.63$	$2.111$
int2	Sliced int8	$39.67$	$9.317$	$40.35$	$7.144$	$38.40$	$10.594$
	Baseline	$47.74$	$3.433$	$56.02$	$2.923$	$54.95$	$2.699$
	${\rm MatQuant}$	$\bf 52.43$	$\bf 3.153$	$\bf 62.32$	$\bf 2.756$	$\bf 61.29$	$\bf 2.474$
int6	Sliced int8	$67.55$	$2.462$	$74.12$	$2.294$	$73.30$	$2.088$
	Baseline	$67.75$	$2.460$	$74.31$	$2.293$	$72.71$	$2.077$
	${\rm MatQuant}$	$67.60$	$2.476$	$74.55$	$2.303$	$72.70$	$2.089$
int3	Sliced int8	$60.23$	$2.913$	$68.57$	$2.565$	$65.29$	$2.441$
	Baseline	$61.75$	$2.678$	$69.9$	$2.43$	$68.82$	$2.197$
	${\rm MatQuant}$	$62.51$	$2.798$	$70.68$	$2.486$	$66.44$	$2.308$

🔼 This table presents a comparison of the performance of Matryoshka Quantization (MatQuant) against baseline methods for quantizing the Gemma-2 2B, 9B, and Mistral 7B models using Quantization Aware Training (QAT). The comparison is made across different bit precisions (int8, int4, int2) and interpolated precisions (int6, int3). MatQuant’s performance is evaluated using average accuracy across downstream tasks and log perplexity on the C4 validation set. The results show that MatQuant achieves comparable or better accuracy than the baselines for various precision levels, demonstrating its effectiveness in multi-scale quantization.
read the caption
Table 2: MatQuantMatQuant{\rm MatQuant}roman_MatQuant with QAT across Gemma-2 2B, 9B and Mistral 7B models. MatQuantMatQuant{\rm MatQuant}roman_MatQuant performs on par with the baseline for int4 and int8 while significantly outperforming it for int2. Even the int3, int6 models obtained for free through interpolation from MatQuantMatQuant{\rm MatQuant}roman_MatQuant perform comparably to the explicitly trained baselines. Task Avg. is average accuracy on the evaluation tasks (↑↑\uparrow↑) while log pplx (perplexity) is computed on C4 validation set (↓↓\downarrow↓).

Data type	Weightings	Gemma-2 2B	Gemma-2 9B	Mistral 7B
		Task Avg.
int8	$(1,1,1)$	$67.42$	$73.97$	$73.46$
	$(1,1,\sqrt{2})$	$67.31$	$73.45$	$73.41$
	$(2,2,1)$	$\bf 67.85$	$74.02$	$\bf 73.82$
	$(\sqrt{2},\sqrt{2},1)$	$67.3$	$\bf 74.33$	$\bf 73.82$
int4	$(1,1,1)$	$66.11$	$73.88$	$73.13$
	$(1,1,\sqrt{2})$	$\bf 66.70$	$73.75$	$73.29$
	$(2,2,1)$	$66.54$	$\bf 74.33$	$\bf 73.5$
	$(\sqrt{2},\sqrt{2},1)$	$66.46$	$74.26$	$72.97$
int2	$(1,1,1)$	$\bf 55.71$	$\bf 68.52$	$65.99$
	$(1,1,\sqrt{2})$	$57.08$	$67.93$	$\bf 66.28$
	$(2,2,1)$	$\bf 55.70$	$66.72$	$63.49$
	$(\sqrt{2},\sqrt{2},1)$	$55.29$	$68.25$	$57.85$

🔼 This table presents an ablation study on the effect of different loss weightings for the three target bit-widths (int8, int4, int2) during Matryoshka Quantization training. It shows how changing the relative importance of the loss functions for each bit-width impacts the final accuracy of the quantized models. The results are presented for the Gemma-2 2B, 9B, and Mistral 7B models. The baseline configuration uses equal weighting (1,1,1). The (0,0,1) configuration represents training only for int2 precision, and it is referred to as ‘Single Precision MatQuant’.
read the caption
Table 3: Design choice ablation for loss re-weighting of the 3 target bit-widths (int8, int4, int2) that MatQuantMatQuant{\rm MatQuant}roman_MatQuant explicitly optimizes. Note that MatQuantMatQuant{\rm MatQuant}roman_MatQuant (0,0,1)001(0,0,1)( 0 , 0 , 1 ) ≡\equiv≡ Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant.

		OmniQuant		QAT
Data type	Config.	Task Avg.	log pplx.	Task Avg.	log pplx.
int8	$[8,4,2]$	$\bf 73.97$	$\bf 2.451$	$74.77$	$2.301$
	$[8,4,8\rightarrow 2]$	$73.40$	$2.467$	$74.72$	$\bf 2.298$
	$[8,4,2,8\rightarrow 2]$	$73.46$	$2.466$	$74.62$	$2.299$
	$[8,4,2,8\rightarrow 4;2]$	$73.32$	$2.466$	$\bf 74.80$	$2.302$
int4	$[8,4,2]$	$\bf 73.88$	$\bf 2.481$	$73.71$	$2.332$
	$[8,4,8\rightarrow 2]$	$73.84$	$2.488$	$73.76$	$\bf 2.328$
	$[8,4,2,8\rightarrow 2]$	$73.01$	$2.495$	$\bf 73.78$	$2.329$
	$[8,4,2,8\rightarrow 4;2]$	$73.12$	$2.518$	$73.48$	$2.330$
int2	$[8,4,2]$	$68.52$	$2.809$	$62.32$	$2.756$
	$[8,4,8\rightarrow 2]$	$69.2$	$2.796$	$61.81$	$\bf 2.740$
	$[8,4,2,8\rightarrow 2]$	$\bf 70.17$	$\bf 2.778$	$\bf 62.51$	$2.746$
	$[8,4,2,8\rightarrow 4;2]$	$69.72$	$2.804$	$62.12$	$2.746$

🔼 This table investigates the impact of co-distillation on Matryoshka Quantization (MatQuant). Co-distillation is a technique where the outputs of a higher-precision model are used as targets to train a lower-precision model. The experiment explores different configurations of distilling int2 and int4 models from an int8 model. The results demonstrate that co-distillation significantly improves the accuracy of the int2 model while having minimal effects on the int4 and int8 models. This highlights the effectiveness of leveraging the nested structure of MatQuant for multi-scale model training.
read the caption
Table 4: Design choice ablations for co-distillation within MatQuantMatQuant{\rm MatQuant}roman_MatQuant. x→y→xy\text{x}\rightarrow\text{y}x → y represents distilling the y-bit model from the x-bit model. We note that the accuracy for int2 has significantly improved while minimally impacting the other bit-widths.

int2	Gemma-2 2B		Gemma-2 9B		Mistral 7B
Method	Task Avg.	log pplx.	Task Avg.	log pplx.	Task Avg.	log pplx.
OmniQuant	$51.33$	$3.835$	$60.24$	$3.292$	$59.74$	$3.931$
S.P. ${\rm MatQuant}$	$\bf 57.38$	$\bf 3.185$	$\bf 68.58$	$2.857$	$\bf 67.36$	$\bf 2.464$
${\rm MatQuant}$	$55.71$	$3.292$	$68.52$	$\bf 2.809$	$65.99$	$2.569$
QAT	$47.74$	$3.433$	$56.02$	$2.923$	$54.95$	$2.699$
S.P. ${\rm MatQuant}$	$\bf 53.18$	$\bf 3.090$	$\bf 62.53$	$\bf 2.706$	$\bf 61.55$	$\bf 2.435$
${\rm MatQuant}$	$52.43$	$3.153$	$62.32$	$2.756$	$61.29$	$2.474$

🔼 Table 5 presents a comparison of the performance of three different quantization methods for int2, int4, and int8 precision levels on the Gemma-2 9B model. The methods are: a baseline approach (where each precision level is trained independently); MatQuant (a multi-scale training technique); and Single Precision MatQuant (a variation of MatQuant that optimizes only for int2 precision). The table shows that Single Precision MatQuant significantly improves the accuracy of the int2 model compared to the baseline but, importantly, leads to substantially lower accuracy for int4 and int8. This suggests that there is a trade-off between optimizing for a single precision level versus multiple levels, and that this tradeoff is important to consider.
read the caption
Table 5: Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant significantly improves upon the baseline for int2 and, at times, outperforms MatQuantMatQuant{\rm MatQuant}roman_MatQuant. Crucially, int8 and int4 performances of Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant experience a significant accuracy decrease (Tables 21 & 22).

Data type	Method	Gemma-2 9B		Mistral 7B
	QAT	Task Avg.	log pplx.	Task Avg.	log pplx.
bfloat16		$74.38$	$2.418$	$73.99$	$2.110$
int8	Baseline	$74.61$	$2.353$	$73.73$	$2.091$
int8	${\rm MatQuant}$	$75.07$	$2.374$	$73.58$	$2.101$
int4	Sliced int8	$73.56$	$2.43$	$71.42$	$2.246$
	Baseline	$72.98$	$2.40$	$71.87$	$2.132$
	${\rm MatQuant}$	$74.11$	$2.436$	$71.5$	$2.166$
int2	Sliced int8	$39.05$	$13.116$	$38.39$	$12.066$
	Baseline	-	-	-	-
	S.P. ${\rm MatQuant}$	$\bf 47.78$	$\bf 3.705$	$34.69$	$7.564$
	${\rm MatQuant}$	$47.17$	$3.837$	$\bf 43.33$	$\bf 3.806$
int6	Sliced int8	$74.56$	$2.358$	$73.71$	$2.094$
	Baseline	$74.65$	$2.357$	$73.72$	$2.093$
	${\rm MatQuant}$	$75.04$	$2.379$	$73.36$	$2.106$
int3	Sliced int8	$64.23$	$2.908$	$39.36$	$4.918$
	Baseline	-	-	-	-
	S.P. ${\rm MatQuant}$	$68.69$	$2.569$	$68.41$	$2.245$
	${\rm MatQuant}$	$66.94$	$2.91$	$59.45$	$2.703$

🔼 Table 6 presents the results of applying Matryoshka Quantization (MatQuant) and Single Precision MatQuant with Quantization Aware Training (QAT) to both Feed Forward Network (FFN) and Attention layers of transformer-based large language models (LLMs). The table compares the performance of different quantization levels (int8, int4, int2, int6, int3) across three model architectures (Gemma-2 2B, Gemma-2 9B and Mistral 7B). It highlights how MatQuant and Single Precision MatQuant stabilize performance at lower bit depths (int2 and int3) where standard QAT often struggles. This showcases the effectiveness of MatQuant in generating models that are both accurate and efficient.
read the caption
Table 6: Extending MatQuantMatQuant{\rm MatQuant}roman_MatQuant with QAT to FFN + Attention parameters. Baseline QAT destabilizes for int2 and int3 but improves significantly through MatQuantMatQuant{\rm MatQuant}roman_MatQuant & Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant.

Data type	Method	Gemma-2 2B
	OmniQuant	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$50.09$	$71.59$	$76.45$	$69.69$	$78.29$	$63.14$	$68.21$
int8	Baseline	$50$	$71.46$	$76.36$	$69.76$	$78.24$	$63.69$	$68.25$
int8	${\rm MatQuant}$	$48.04$	$71.8$	$75.78$	$67.64$	$78.07$	$63.22$	$67.42$
int4	Sliced int8	$41.81$	$66.2$	$71.35$	$62.64$	$75.95$	$59.91$	$62.98$
	Baseline	$48.46$	$70.96$	$74.22$	$67.66$	$77.26$	$63.61$	$67.03$
	${\rm MatQuant}$	$45.65$	$70.29$	$74.8$	$66.07$	$77.58$	$62.27$	$66.11$
int2	Sliced int8	$23.81$	$23.53$	$53.06$	$24.78$	$51.8$	$49.09$	$37.68$
	Baseline	$31.31$	$53.58$	$62.2$	$40.78$	$66.05$	$54.06$	$51.33$
	${\rm MatQuant}$	$34.39$	$59.64$	$62.69$	$52.11$	$69.86$	$55.56$	$55.71$
int6	Sliced int8	$48.55$	$71.25$	$75.87$	$69.18$	$78.35$	$62.75$	$67.66$
	Baseline	$49.32$	$71.76$	$76.48$	$69.52$	$78.56$	$62.75$	$68.06$
	${\rm MatQuant}$	$47.1$	$71.46$	$76.02$	$67.47$	$77.91$	$63.61$	$67.26$
int3	Sliced int8	$23.21$	$34.43$	$58.2$	$30.48$	$56.69$	$49.01$	$42$
	Baseline	$46.25$	$68.64$	$72.97$	$62.24$	$76.06$	$60.06$	$64.37$
	${\rm MatQuant}$	$44.45$	$68.56$	$69.11$	$62.28$	$75.95$	$62.59$	$63.82$

🔼 This table presents the downstream evaluation results of Matryoshka Quantization (MatQuant) when it is applied to the OmniQuant method on the Gemma-2 2B model. Downstream evaluation metrics such as accuracy across various tasks (ARC-c, ARC-e, BoolQ, HellaSwag, PIQA, and Winogrande) are shown for different quantization levels (int8, int4, int2, int6, int3). The results are compared against baseline OmniQuant and sliced int8 models for a comprehensive comparison. It highlights MatQuant’s performance across various bit precisions and its ability to generate multiple quantized models from a single trained model.
read the caption
Table 7: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant when applied to OmniQuant on Gemma-2 2B.

Data type	Method	Gemma-2 9B
	OmniQuant	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$58.96$	$77.57$	$83.33$	$77.31$	$81.12$	$67.96$	$74.38$
int8	Baseline	$59.47$	$77.31$	$83.94$	$77.35$	$81.39$	$68.11$	$74.59$
int8	${\rm MatQuant}$	$58.11$	$78.03$	$83.27$	$76.17$	$81.18$	$67.09$	$73.97$
int4	Sliced int8	$55.97$	$75.04$	$81.19$	$73.81$	$80.52$	$66.61$	$72.19$
	Baseline	$58.79$	$78.37$	$83.55$	$76.71$	$81.45$	$67.09$	$74.33$
	${\rm MatQuant}$	$57.25$	$77.36$	$84.86$	$75.52$	$81.5$	$66.77$	$73.88$
int2	Sliced int8	$23.21$	$24.92$	$38.13$	$25.37$	$51.36$	$51.54$	$35.75$
	Baseline	$39.16$	$63.43$	$72.11$	$52.24$	$72.63$	$61.88$	$60.24$
	${\rm MatQuant}$	$48.72$	$72.18$	$79.2$	$68.11$	$76.17$	$66.77$	$68.52$
int6	Sliced int8	$59.04$	$77.53$	$84.68$	$77.1$	$81.23$	$68.11$	$74.61$
	Baseline	$59.22$	$77.27$	$83.21$	$77.1$	$81.12$	$67.48$	$74.23$
	${\rm MatQuant}$	$58.87$	$78.03$	$83.61$	$76.18$	$81.45$	$67.09$	$74.21$
int3	Sliced int8	$35.84$	$57.32$	$67.61$	$48.58$	$68.61$	$56.59$	$55.76$
	Baseline	$57.17$	$77.06$	$83.79$	$74.45$	$80.36$	$66.54$	$73.23$
	${\rm MatQuant}$	$55.46$	$76.14$	$84.04$	$74.49$	$80.14$	$67.32$	$72.93$

🔼 This table presents the downstream evaluation results for the Matryoshka Quantization (MatQuant) technique when used with OmniQuant on the Gemma-2 9B model. It shows the performance of MatQuant across different quantization levels (int8, int4, int2, int6, int3) on various downstream tasks such as ARC-c, ARC-e, BoolQ, HellaSwag, PIQA, and Winogrande. The results are compared against baselines representing standard OmniQuant without MatQuant and also those derived by slicing an int8 quantized model. This allows for a comparison of MatQuant’s multi-scale training approach against traditional single-precision quantization methods and to evaluate how it performs with interpolated bit-widths.
read the caption
Table 8: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant when applied to OmniQuant on Gemma-2 9B.

Data type	Method	Mistral 7B
	OmniQuant	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$49.57$	$73.74$	$84.4$	$80.61$	$81.18$	$74.43$	$73.99$
int8	Baseline	$49.23$	$73.19$	$83.88$	$80.41$	$81.39$	$74.51$	$73.77$
int8	${\rm MatQuant}$	$48.04$	$73.44$	$84.13$	$79.37$	$81.12$	$74.66$	$73.46$
int4	Sliced int8	$27.65$	$46.72$	$49.17$	$36.88$	$64.09$	$55.01$	$46.59$
	Baseline	$49.23$	$73.23$	$83.94$	$79.9$	$81.34$	$74.11$	$73.62$
	${\rm MatQuant}$	$48.21$	$72.69$	$83.49$	$78.82$	$81.12$	$74.43$	$73.13$
int2	Sliced int8	$23.72$	$25.29$	$43.21$	$25.45$	$50.49$	$49.33$	$36.25$
	Baseline	$36.69$	$61.36$	$70.06$	$57.47$	$70.67$	$62.19$	$59.74$
	${\rm MatQuant}$	$41.38$	$67.42$	$71.62$	$71.98$	$77.86$	$65.67$	$65.99$
int6	Sliced int8	$48.98$	$72.01$	$83.46$	$79.95$	$81.72$	$74.9$	$73.5$
	Baseline	$50.26$	$73.65$	$84.04$	$80.55$	$81.66$	$74.43$	$74.1$
	${\rm MatQuant}$	$48.46$	$72.98$	$84.07$	$79.64$	$81.18$	$75.22$	$73.59$
int3	Sliced int8	$22.78$	$24.66$	$37.86$	$24.12$	$49.24$	$48.93$	$34.6$
	Baseline	$46.33$	$70.71$	$82.72$	$77.74$	$80.74$	$71.82$	$71.68$
	${\rm MatQuant}$	$45.65$	$71.21$	$80.43$	$78.31$	$81.07$	$72.61$	$71.55$

🔼 This table presents the downstream task evaluation results for the Matryoshka Quantization (MatQuant) method when used with OmniQuant on the Mistral 7B model. It shows the performance (accuracy and log-perplexity) for different quantization levels (int8, int4, int2, int6, int3). For each precision, MatQuant’s performance is compared against a baseline (explicitly trained for that precision) and the results from slicing a higher precision model (sliced int8). This allows for a comparison of the effectiveness of the multi-scale training method against training separate models at different precisions.
read the caption
Table 9: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant when applied to OmniQuant on Mistral 7B.

Data type	Method	Gemma-2 2B
	QAT	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$50.09$	$71.59$	$76.45$	$69.69$	$78.29$	$63.14$	$68.21$
int8	Baseline	$47.78$	$70.66$	$75.08$	$69.92$	$78.35$	$65.11$	$67.82$
int8	${\rm MatQuant}$	$46.25$	$71.21$	$75.6$	$69.97$	$78.4$	$64.64$	$67.68$
int4	Sliced int8	$46.08$	$69.36$	$75.78$	$68.05$	$78.18$	$65.75$	$67.2$
	Baseline	$46.16$	$71.59$	$73.73$	$68.72$	$78.62$	$63.38$	$67.03$
	${\rm MatQuant}$	$44.37$	$70.45$	$75.81$	$68.43$	$78.35$	$64.88$	$67.05$
int2	Sliced int8	$25.6$	$26.3$	$57.98$	$25.82$	$52.12$	$50.2$	$39.67$
	Baseline	$24.66$	$43.22$	$62.17$	$38.39$	$64.42$	$53.59$	$47.74$
	${\rm MatQuant}$	$28.24$	$51.73$	$64.19$	$46.76$	$68.66$	$55.01$	$52.43$
int6	Sliced int8	$47.78$	$70.79$	$74.25$	$69.73$	$77.64$	$65.11$	$67.55$
	Baseline	$47.7$	$70.88$	$74.92$	$69.72$	$78.07$	$65.19$	$67.75$
	${\rm MatQuant}$	$46.5$	$70.71$	$75.72$	$69.69$	$78.02$	$64.96$	$67.6$
int3	Sliced int8	$38.74$	$63.13$	$65.57$	$58.86$	$74.81$	$60.3$	$60.23$
	Baseline	$39.68$	$65.28$	$67.03$	$62.68$	$77.04$	$58.8$	$61.75$
	${\rm MatQuant}$	$38.65$	$67.34$	$70.49$	$61.47$	$75.41$	$61.72$	$62.51$

🔼 This table presents the downstream evaluation results for the Matryoshka Quantization (MatQuant) method when applied to Quantization Aware Training (QAT) on the Gemma-2 2B model. It compares the performance of MatQuant across different bit-widths (int8, int4, int2, int6, and int3) against baseline models trained with QAT at each bit-width. The evaluation metrics include accuracy on various downstream tasks (ARC-c, ARC-e, BoolQ, HellaSwag, PIQA, and Winogrande) and average perplexity on the C4 validation set. It also includes results for models derived from MatQuant via interpolation (sliced int8) to show the adaptability and efficiency of MatQuant in generating models that perform well at different bit-widths without retraining.
read the caption
Table 10: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant when applied to QAT on Gemma-2 2B.

Data type	Method	Gemma-2 9B
	QAT	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$58.96$	$77.57$	$83.33$	$77.31$	$81.12$	$67.96$	$74.38$
int8	Baseline	$58.11$	$75.38$	$80.12$	$78.7$	$81.5$	$71.19$	$74.17$
int8	${\rm MatQuant}$	$58.19$	$76.18$	$81.5$	$79.57$	$82.15$	$71.03$	$74.77$
int4	Sliced int8	$57.42$	$75.08$	$78.1$	$76.97$	$81.23$	$70.72$	$73.25$
	Baseline	$56.91$	$75.42$	$75.38$	$78.06$	$81.39$	$72.38$	$73.26$
	${\rm MatQuant}$	$57.94$	$76.64$	$75.2$	$78.71$	$81.66$	$72.14$	$73.71$
int2	Sliced int8	$23.89$	$27.61$	$57.95$	$30.16$	$54.68$	$47.83$	$40.35$
	Baseline	$33.45$	$55.43$	$62.26$	$54.8$	$70.51$	$59.67$	$56.02$
	${\rm MatQuant}$	$39.85$	$65.66$	$65.93$	$64.08$	$75.68$	$62.75$	$62.32$
int6	Sliced int8	$57.85$	$75.13$	$80.67$	$78.63$	$81.56$	$70.88$	$74.12$
	Baseline	$57.94$	$76.14$	$79.63$	$78.93$	$82.1$	$71.11$	$74.31$
	${\rm MatQuant}$	$58.02$	$75.63$	$81.31$	$79.43$	$81.66$	$71.27$	$74.55$
int3	Sliced int8	$50$	$68.1$	$75.2$	$71.31$	$79.43$	$67.4$	$68.57$
	Baseline	$53.07$	$75.04$	$66.61$	$74.94$	$80.03$	$69.69$	$69.9$
	${\rm MatQuant}$	$51.62$	$71.93$	$78.78$	$73.99$	$80.14$	$67.64$	$70.68$

🔼 This table displays the performance of Matryoshka Quantization (MatQuant) when integrated with Quantization Aware Training (QAT) on the Gemma-2 9B model. It compares the average accuracy across several downstream tasks for various quantization bit-widths (int8, int4, int2, int6, int3). The table includes results for MatQuant, baselines (explicitly trained models for each precision), and models obtained by slicing an int8-quantized model. This allows for a comparison of MatQuant’s multi-scale training approach against independently trained models of different precisions, highlighting MatQuant’s ability to efficiently create multiple models from a single trained model.
read the caption
Table 11: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant when applied to QAT on Gemma-2 9B.

Data type	Method	Mistral 7B
	QAT	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$49.57$	$73.74$	$84.4$	$80.61$	$81.18$	$74.43$	$73.99$
int8	Baseline	$48.89$	$71.63$	$82.42$	$81.69$	$81.18$	$75.06$	$73.48$
int8	${\rm MatQuant}$	$46.76$	$70.37$	$82.51$	$79.73$	$80.9$	$74.19$	$72.41$
int4	Sliced int8	$47.18$	$70.41$	$80.37$	$79.84$	$80.25$	$72.93$	$71.83$
	Baseline	$47.27$	$70.62$	$81.28$	$78.95$	$81.12$	$73.56$	$72.13$
	${\rm MatQuant}$	$45.65$	$68.64$	$82.02$	$79$	$81.07$	$73.4$	$71.63$
int2	Sliced int8	$25.34$	$26.47$	$54.95$	$25.18$	$48.48$	$49.96$	$38.4$
	Baseline	$29.78$	$48.23$	$64.5$	$55.11$	$70.84$	$61.25$	$54.95$
	${\rm MatQuant}$	$34.3$	$55.09$	$71.83$	$65.89$	$75.52$	$65.11$	$61.29$
int6	Sliced int8	$48.21$	$71.51$	$82.42$	$81.67$	$81.72$	$74.27$	$73.3$
	Baseline	$47.7$	$71.3$	$82.23$	$79.84$	$80.79$	$74.43$	$72.71$
	${\rm MatQuant}$	$47.53$	$71$	$81.9$	$79.73$	$81.28$	$74.74$	$72.7$
int3	Sliced int8	$40.1$	$61.49$	$72.91$	$68.72$	$77.97$	$70.56$	$65.29$
	Baseline	$44.54$	$67.97$	$73.98$	$76.31$	$79.65$	$70.48$	$68.82$
	${\rm MatQuant}$	$38.82$	$62.42$	$77.74$	$71.1$	$78.07$	$70.48$	$66.44$

🔼 This table presents the downstream evaluation results for the Matryoshka Quantization (MatQuant) technique when applied to Quantization Aware Training (QAT) using the Mistral 7B model. It shows the performance of MatQuant across multiple bit-widths (int8, int4, int2, int6, int3) on various downstream tasks, comparing its results to both a standard baseline and a baseline where lower-precision models are created by simply slicing the bits from an int8-quantized model. The metrics used include average accuracy across the downstream tasks and log perplexity.
read the caption
Table 12: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant when applied to QAT on Mistral 7B.

		Gemma-2 2B
Data type	Weightings	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
int8	$(1,1,1)$	$48.04$	$71.8$	$75.78$	$67.64$	$78.07$	$63.22$	$67.42$
	$(1\sqrt{2},\sqrt{2})$	$47.35$	$71.34$	$75.66$	$67.99$	$78.07$	$63.38$	$67.3$
	$(\sqrt{2},1,\sqrt{2})$	$47.44$	$72.43$	$76.02$	$67.45$	$78.02$	$63.85$	$67.54$
	$(1,1\sqrt{2})$	$47.7$	$71.89$	$75.63$	$67.21$	$78.07$	$63.38$	$67.31$
	$(2,2,1)$	$48.38$	$72.31$	$76.3$	$68.32$	$78.35$	$63.46$	$67.85$
	$(\sqrt{2},2,1)$	$48.46$	$71.84$	$75.93$	$68.35$	$77.91$	$63.14$	$67.6$
	$(2,\sqrt{2},1)$	$47.95$	$71.72$	$75.26$	$68.13$	$78.07$	$62.75$	$67.31$
	$(\sqrt{2},\sqrt{2},1)$	$47.35$	$71.34$	$75.66$	$67.99$	$78.07$	$63.38$	$67.3$
int4	$(1,1,1)$	$45.65$	$70.29$	$74.8$	$66.07$	$77.58$	$62.27$	$66.11$
	$(1\sqrt{2},\sqrt{2})$	$46.33$	$70.92$	$73.7$	$67.67$	$77.26$	$62.9$	$66.46$
	$(\sqrt{2},1,\sqrt{2})$	$46.42$	$70.96$	$74.71$	$65.78$	$77.58$	$63.14$	$66.43$
	$(1,1\sqrt{2})$	$45.56$	$71.55$	$75.75$	$66.18$	$77.48$	$63.69$	$66.7$
	$(2,2,1)$	$46.84$	$70.88$	$74.92$	$66.48$	$77.91$	$62.19$	$66.54$
	$(\sqrt{2},2,1)$	$47.35$	$71.68$	$72.69$	$66.79$	$77.26$	$63.38$	$66.52$
	$(2,\sqrt{2},1)$	$45.9$	$70.83$	$75.11$	$66.97$	$77.37$	$62.27$	$66.41$
	$(\sqrt{2},\sqrt{2},1)$	$46.33$	$70.92$	$73.7$	$67.67$	$77.26$	$62.9$	$66.46$
int2	$(1,1,1)$	$34.39$	$59.64$	$62.69$	$52.11$	$69.86$	$55.56$	$55.71$
	$(1\sqrt{2},\sqrt{2})$	$32.76$	$56.99$	$63.46$	$51.99$	$70.29$	$56.27$	$55.29$
	$(\sqrt{2},1,\sqrt{2})$	$35.07$	$62.04$	$65.78$	$54.26$	$71.65$	$56.27$	$57.51$
	$(1,1\sqrt{2})$	$34.22$	$60.4$	$64.98$	$54.3$	$71.38$	$57.22$	$57.08$
	$(2,2,1)$	$34.47$	$57.95$	$63.94$	$51.84$	$69.75$	$56.27$	$55.7$
	$(\sqrt{2},2,1)$	$33.45$	$57.49$	$65.02$	$52.22$	$70.4$	$55.64$	$55.7$
	$(2,\sqrt{2},1)$	$34.04$	$58.84$	$65.11$	$51.77$	$70.89$	$57.14$	$56.3$
	$(\sqrt{2},\sqrt{2},1)$	$32.76$	$56.99$	$63.46$	$51.99$	$70.29$	$56.27$	$55.29$
int6	$(1,1,1)$	$47.1$	$71.46$	$76.02$	$67.47$	$77.91$	$63.61$	$67.26$
	$(1\sqrt{2},\sqrt{2})$	$47.44$	$71.42$	$74.95$	$67.85$	$77.86$	$63.3$	$67.14$
	$(\sqrt{2},1,\sqrt{2})$	$47.61$	$71.89$	$75.9$	$67.37$	$78.24$	$63.77$	$67.46$
	$(1,1\sqrt{2})$	$47.78$	$71.63$	$75.47$	$67.2$	$77.86$	$63.61$	$67.26$
	$(2,2,1)$	$48.55$	$72.69$	$76.3$	$68.02$	$78.67$	$63.85$	$68.01$
	$(\sqrt{2},2,1)$	$48.29$	$71.76$	$75.72$	$68.42$	$78.02$	$63.38$	$67.6$
	$(2,\sqrt{2},1)$	$48.38$	$71.51$	$75.84$	$68.24$	$78.18$	$63.85$	$67.67$
	$(\sqrt{2},\sqrt{2},1)$	$47.44$	$71.42$	$74.95$	$67.85$	$77.86$	$63.3$	$67.14$
int3	$(1,1,1)$	$44.45$	$68.56$	$69.11$	$62.28$	$75.95$	$62.59$	$63.82$
	$(1\sqrt{2},\sqrt{2})$	$43.17$	$68.73$	$64.74$	$61.31$	$76.39$	$61.48$	$62.64$
	$(\sqrt{2},1,\sqrt{2})$	$41.98$	$68.6$	$70.34$	$61.95$	$75.9$	$63.3$	$63.68$
	$(1,1\sqrt{2})$	$41.64$	$66.71$	$71.62$	$61.94$	$76.01$	$61.09$	$63.17$
	$(2,2,1)$	$41.98$	$68.35$	$68.41$	$63.74$	$76.17$	$60.77$	$63.24$
	$(\sqrt{2},2,1)$	$42.66$	$66.54$	$70.46$	$63.61$	$75.63$	$62.98$	$63.65$
	$(2,\sqrt{2},1)$	$43.17$	$66.71$	$60.03$	$62.71$	$76.77$	$61.64$	$61.84$
	$(\sqrt{2},\sqrt{2},1)$	$43.17$	$68.73$	$64.74$	$61.31$	$76.39$	$61.48$	$62.64$

🔼 This table presents the downstream evaluation results for the Gemma-2 2B model when using Matryoshka Quantization (MatQuant) with OmniQuant. It shows the impact of different loss weightings applied during MatQuant training on the accuracy of the quantized model across various bit-widths (int8, int4, int2, int6, int3). The loss weightings represent the relative importance given to the different bit-widths in the optimization process, allowing the exploration of different trade-offs between the accuracy levels across precisions. The results are shown for different weight combinations, allowing comparison of performance based on these weighting schemes.
read the caption
Table 13: Tables presents the downstream evaluation results on Gemma-2 2B for MatQuantMatQuant{\rm MatQuant}roman_MatQuant loss reweighting when applied to OmniQuant. Weightings: (x,y,z)→(λ8,λ4,λ2)→𝑥𝑦𝑧subscript𝜆8subscript𝜆4subscript𝜆2(x,y,z)\rightarrow(\lambda_{8},\lambda_{4},\lambda_{2})( italic_x , italic_y , italic_z ) → ( italic_λ start_POSTSUBSCRIPT 8 end_POSTSUBSCRIPT , italic_λ start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT , italic_λ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) (from Equation 7).

		Gemma-2 9B
Data type	Weightings	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
int8	$(1,1,1)$	$58.11$	$78.03$	$83.27$	$76.17$	$81.18$	$67.09$	$73.97$
	$(1\sqrt{2},\sqrt{2})$	$57.68$	$77.4$	$83.73$	$76.1$	$81.18$	$67.64$	$73.95$
	$(\sqrt{2},1,\sqrt{2})$	$58.11$	$77.86$	$81.04$	$76$	$81.18$	$67.09$	$73.55$
	$(1,1\sqrt{2})$	$56.91$	$77.1$	$82.39$	$75.93$	$81.18$	$67.17$	$73.45$
	$(2,2,1)$	$58.79$	$77.48$	$82.66$	$76.55$	$81.23$	$67.4$	$74.02$
	$(\sqrt{2},2,1)$	$58.53$	$77.31$	$82.63$	$76.54$	$80.96$	$67.56$	$73.92$
	$(2,\sqrt{2},1)$	$58.62$	$77.27$	$84.31$	$76.54$	$81.34$	$66.85$	$74.16$
	$(\sqrt{2},\sqrt{2},1)$	$59.13$	$78.07$	$84.16$	$76.46$	$80.9$	$67.25$	$74.33$
int4	$(1,1,1)$	$57.25$	$77.36$	$84.86$	$75.52$	$81.5$	$66.77$	$73.88$
	$(1\sqrt{2},\sqrt{2})$	$56.74$	$77.74$	$85.08$	$75.5$	$80.85$	$66.85$	$73.79$
	$(\sqrt{2},1,\sqrt{2})$	$57.42$	$78.28$	$82.51$	$75.97$	$81.34$	$67.56$	$73.85$
	$(1,1\sqrt{2})$	$57.59$	$77.82$	$84.28$	$75.32$	$81.12$	$66.38$	$73.75$
	$(2,2,1)$	$58.62$	$78.28$	$83.67$	$76.01$	$81.5$	$67.88$	$74.33$
	$(\sqrt{2},2,1)$	$58.19$	$77.82$	$83.91$	$76.62$	$81.99$	$67.72$	$74.37$
	$(2,\sqrt{2},1)$	$58.28$	$78.16$	$84.53$	$76.41$	$81.72$	$67.09$	$74.36$
	$(\sqrt{2},\sqrt{2},1)$	$57.94$	$78.11$	$84.98$	$76.5$	$81.01$	$67.01$	$74.26$
int2	$(1,1,1)$	$48.72$	$72.18$	$79.2$	$68.11$	$76.17$	$66.77$	$68.52$
	$(1\sqrt{2},\sqrt{2})$	$49.83$	$73.91$	$78.75$	$67.27$	$77.2$	$66.46$	$68.9$
	$(\sqrt{2},1,\sqrt{2})$	$48.55$	$74.24$	$81.5$	$68.44$	$76.5$	$65.9$	$69.19$
	$(1,1\sqrt{2})$	$48.29$	$72.94$	$74.74$	$68.34$	$77.58$	$65.67$	$67.93$
	$(2,2,1)$	$46.76$	$73.27$	$71.96$	$67.98$	$76.77$	$63.61$	$66.72$
	$(\sqrt{2},2,1)$	$46.76$	$73.7$	$77.65$	$67.01$	$77.58$	$65.98$	$68.11$
	$(2,\sqrt{2},1)$	$46.76$	$72.35$	$75.35$	$67.51$	$76.39$	$67.56$	$67.65$
	$(\sqrt{2},\sqrt{2},1)$	$46.59$	$72.6$	$79.3$	$67.58$	$77.69$	$65.75$	$68.25$
int6	$(1,1,1)$	$58.87$	$78.03$	$83.61$	$76.18$	$81.45$	$67.09$	$74.21$
	$(1\sqrt{2},\sqrt{2})$	$57.51$	$77.53$	$83.55$	$75.98$	$80.9$	$67.17$	$73.77$
	$(\sqrt{2},1,\sqrt{2})$	$58.79$	$77.82$	$81.38$	$76.21$	$81.07$	$67.72$	$73.83$
	$(1,1\sqrt{2})$	$57.34$	$77.23$	$82.57$	$75.89$	$81.12$	$67.17$	$73.55$
	$(2,2,1)$	$59.04$	$77.4$	$82.66$	$76.55$	$81.56$	$68.03$	$74.21$
	$(\sqrt{2},2,1)$	$59.22$	$77.65$	$82.17$	$76.62$	$81.23$	$67.8$	$74.11$
	$(2,\sqrt{2},1)$	$58.36$	$77.82$	$83.79$	$76.47$	$81.23$	$67.25$	$74.15$
	$(\sqrt{2},\sqrt{2},1)$	$59.3$	$78.37$	$84.5$	$76.57$	$80.85$	$67.4$	$74.5$
int3	$(1,1,1)$	$55.46$	$76.14$	$84.04$	$74.49$	$80.14$	$67.32$	$72.93$
	$(1\sqrt{2},\sqrt{2})$	$56.23$	$76.05$	$82.6$	$74.85$	$80.9$	$67.01$	$72.94$
	$(\sqrt{2},1,\sqrt{2})$	$56.4$	$77.86$	$80.64$	$75.11$	$79.87$	$68.51$	$73.06$
	$(1,1\sqrt{2})$	$55.63$	$76.05$	$82.39$	$74.21$	$80.3$	$67.17$	$72.62$
	$(2,2,1)$	$55.2$	$76.56$	$84.19$	$74.87$	$80.2$	$67.72$	$73.12$
	$(\sqrt{2},2,1)$	$54.44$	$75.63$	$80.55$	$74.97$	$80.96$	$67.72$	$72.38$
	$(2,\sqrt{2},1)$	$56.14$	$75.67$	$83.33$	$74.96$	$80.52$	$67.72$	$73.06$
	$(\sqrt{2},\sqrt{2},1)$	$56.31$	$77.4$	$83.24$	$75.62$	$80.41$	$66.54$	$73.25$

🔼 This table presents the downstream evaluation results of the Matryoshka Quantization (MatQuant) method on the Gemma-2 9B model when using OmniQuant. The experiment focuses on the impact of different weightings applied during training. The weightings control the relative importance of achieving high accuracy at different bit precisions (int8, int4, int2), which are determined by the λ parameters in Equation 7 of the paper. Each row represents a different set of weightings, showing how the average accuracy across multiple downstream tasks changes. The table allows comparison of MatQuant’s performance under various weighting schemes to a baseline (no weighting) and to versions where precision levels are derived by simply slicing from the int8 weights.
read the caption
Table 14: Tables presents the downstream evaluation results on Gemma-2 9B for MatQuantMatQuant{\rm MatQuant}roman_MatQuant loss reweighting when applied to OmniQuant. Weightings: (x,y,z)→(λ8,λ4,λ2)→𝑥𝑦𝑧subscript𝜆8subscript𝜆4subscript𝜆2(x,y,z)\rightarrow(\lambda_{8},\lambda_{4},\lambda_{2})( italic_x , italic_y , italic_z ) → ( italic_λ start_POSTSUBSCRIPT 8 end_POSTSUBSCRIPT , italic_λ start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT , italic_λ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) (from Equation 7).

		Mistral 7B
Data type	Weightings	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
int8	$(1,1,1)$	$48.04$	$73.44$	$84.13$	$79.37$	$81.12$	$74.66$	$73.46$
	$(1\sqrt{2},\sqrt{2})$	$48.46$	$73.19$	$84.28$	$79.19$	$81.12$	$74.74$	$73.5$
	$(\sqrt{2},1,\sqrt{2})$	$47.95$	$73.4$	$84.46$	$79.11$	$81.34$	$74.51$	$73.46$
	$(1,1\sqrt{2})$	$48.21$	$73.02$	$84.34$	$79.03$	$81.28$	$74.59$	$73.41$
	$(2,2,1)$	$49.06$	$73.48$	$84.74$	$79.73$	$81.56$	$74.35$	$73.82$
	$(\sqrt{2},2,1)$	$49.06$	$73.57$	$84.56$	$79.64$	$81.39$	$74.27$	$73.75$
	$(2,\sqrt{2},1)$	$48.98$	$73.95$	$84.50$	$79.60$	$81.61$	$74.90$	$73.92$
	$(\sqrt{2},\sqrt{2},1)$	$48.98$	$73.86$	$84.56$	$79.55$	$81.23$	$74.74$	$73.82$
int4	$(1,1,1)$	$48.21$	$72.69$	$83.49$	$78.82$	$81.12$	$74.43$	$73.13$
	$(1\sqrt{2},\sqrt{2})$	$49.15$	$72.81$	$83.39$	$78.71$	$80.79$	$74.66$	$73.25$
	$(\sqrt{2},1,\sqrt{2})$	$47.95$	$72.43$	$83.43$	$79.24$	$81.01$	$74.03$	$73.01$
	$(1,1\sqrt{2})$	$48.46$	$73.44$	$84.07$	$78.9$	$81.01$	$73.88$	$73.29$
	$(2,2,1)$	$49.15$	$72.81$	$83.88$	$79.8$	$81.88$	$73.48$	$73.5$
	$(\sqrt{2},2,1)$	$48.89$	$72.69$	$82.72$	$79.53$	$81.66$	$73.88$	$73.23$
	$(2,\sqrt{2},1)$	$47.87$	$72.05$	$83$	$79.56$	$81.23$	$74.27$	$73$
	$(\sqrt{2},\sqrt{2},1)$	$48.29$	$72.47$	$82.84$	$79.52$	$81.07$	$73.64$	$72.97$
int2	$(1,1,1)$	$41.38$	$67.42$	$71.62$	$71.98$	$77.86$	$65.67$	$65.99$
	$(1\sqrt{2},\sqrt{2})$	$40.78$	$66.2$	$73.61$	$72.68$	$77.75$	$67.4$	$66.4$
	$(\sqrt{2},1,\sqrt{2})$	$40.36$	$67.09$	$75.35$	$72.46$	$77.48$	$65.9$	$66.44$
	$(1,1\sqrt{2})$	$40.36$	$67.17$	$74.83$	$71.64$	$77.53$	$66.14$	$66.28$
	$(2,2,1)$	$37.2$	$62.46$	$67.74$	$70.29$	$76.55$	$66.69$	$63.49$
	$(\sqrt{2},2,1)$	$37.29$	$64.35$	$61.1$	$68.88$	$74.86$	$65.19$	$61.94$
	$(2,\sqrt{2},1)$	$39.68$	$65.24$	$68.93$	$66.64$	$75.19$	$64.09$	$63.29$
	$(\sqrt{2},\sqrt{2},1)$	$34.56$	$61.24$	$60.61$	$58.07$	$72.63$	$59.98$	$57.85$
int6	$(1,1,1)$	$48.46$	$72.98$	$84.07$	$79.64$	$81.18$	$75.22$	$73.59$
	$(1\sqrt{2},\sqrt{2})$	$49.06$	$73.44$	$84.59$	$79.51$	$81.28$	$74.74$	$73.77$
	$(\sqrt{2},1,\sqrt{2})$	$47.95$	$73.48$	$84.43$	$79.28$	$81.45$	$75.14$	$73.62$
	$(1,1\sqrt{2})$	$48.38$	$72.94$	$84.34$	$79.15$	$81.18$	$74.59$	$73.43$
	$(2,2,1)$	$48.46$	$72.94$	$84.13$	$79.89$	$81.5$	$74.9$	$73.64$
	$(\sqrt{2},2,1)$	$48.81$	$73.48$	$84.34$	$79.67$	$81.34$	$74.9$	$73.76$
	$(2,\sqrt{2},1)$	$49.4$	$73.65$	$84.4$	$79.68$	$81.28$	$74.74$	$73.86$
	$(\sqrt{2},\sqrt{2},1)$	$49.23$	$73.57$	$84.43$	$79.55$	$81.12$	$74.66$	$73.76$
int3	$(1,1,1)$	$45.65$	$71.21$	$80.43$	$78.31$	$81.07$	$72.61$	$71.55$
	$(1\sqrt{2},\sqrt{2})$	$47.7$	$72.05$	$82.81$	$78.74$	$81.12$	$72.77$	$72.53$
	$(\sqrt{2},1,\sqrt{2})$	$46.33$	$72.43$	$81.8$	$79.03$	$82.1$	$73.4$	$72.51$
	$(1,1\sqrt{2})$	$45.99$	$71.09$	$80.73$	$78.77$	$80.85$	$72.53$	$71.66$
	$(2,2,1)$	$47.95$	$73.36$	$82.57$	$79.31$	$81.39$	$74.9$	$73.25$
	$(\sqrt{2},2,1)$	$44.45$	$69.7$	$82.11$	$77.68$	$80.2$	$71.74$	$70.98$
	$(2,\sqrt{2},1)$	$46.84$	$72.73$	$80.95$	$78.79$	$81.56$	$73.01$	$72.31$
	$(\sqrt{2},\sqrt{2},1)$	$47.01$	$71.59$	$81.96$	$78.89$	$81.39$	$72.45$	$72.22$

🔼 This table presents the results of a downstream evaluation on the Mistral 7B model. The experiment involves Matryoshka Quantization (MatQuant) with OmniQuant, focusing on the effect of different loss weightings during training. The loss weights (λ8, λ4, λ2) control the relative importance of the int8, int4, and int2 precision levels during optimization. The table shows the performance metrics (accuracy across various downstream tasks) for different precision levels (int8, int4, int2, int6, int3) resulting from varying these loss weights, providing insights into the trade-offs involved in multi-scale quantization.
read the caption
Table 15: Tables presents the downstream evaluation results on Mistral 7B for MatQuantMatQuant{\rm MatQuant}roman_MatQuant loss reweighting when applied to OmniQuant. Weightings: (x,y,z)→(λ8,λ4,λ2)→𝑥𝑦𝑧subscript𝜆8subscript𝜆4subscript𝜆2(x,y,z)\rightarrow(\lambda_{8},\lambda_{4},\lambda_{2})( italic_x , italic_y , italic_z ) → ( italic_λ start_POSTSUBSCRIPT 8 end_POSTSUBSCRIPT , italic_λ start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT , italic_λ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) (from Equation 7).

Data type	Config.	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average	log pplx.
	OmniQuant	Gemma-2 9B
int8	$[8,4,8\rightarrow 2]$	$57.59$	$77.27$	$81.83$	$75.48$	$81.01$	$67.25$	$73.4$	$2.467$
	$[8,4,2,8\rightarrow 2]$	$57.17$	$77.36$	$82.2$	$75.82$	$80.96$	$67.25$	$73.46$	$2.466$
	$[8,4,2,8\rightarrow 4;2]$	$56.4$	$77.82$	$82.32$	$75.02$	$80.63$	$67.72$	$73.32$	$2.466$
int4	$[8,4,8\rightarrow 2]$	$57.68$	$78.45$	$82.97$	$75.5$	$80.85$	$67.56$	$73.84$	$2.488$
	$[8,4,2,8\rightarrow 2]$	$57.51$	$77.61$	$80.46$	$74.74$	$81.12$	$66.61$	$73.01$	$2.495$
	$[8,4,2,8\rightarrow 4;2]$	$56.57$	$77.99$	$82.54$	$74.77$	$80.58$	$66.3$	$73.12$	$2.518$
int2	$[8,4,8\rightarrow 2]$	$48.81$	$74.03$	$81.65$	$68.1$	$77.48$	$65.11$	$69.2$	$2.796$
	$[8,4,2,8\rightarrow 2]$	$49.15$	$75.34$	$83.12$	$68.79$	$77.64$	$67.01$	$70.17$	$2.778$
	$[8,4,2,8\rightarrow 4;2]$	$49.83$	$75.04$	$79.79$	$68.38$	$77.86$	$67.4$	$69.72$	$2.804$
int6	$[8,4,8\rightarrow 2]$	$57.42$	$77.19$	$81.87$	$75.42$	$81.01$	$67.8$	$73.45$	$2.468$
	$[8,4,2,8\rightarrow 2]$	$57.51$	$77.48$	$82.32$	$75.88$	$81.07$	$66.61$	$73.48$	$2.467$
	$[8,4,2,8\rightarrow 4;2]$	$56.4$	$78.03$	$82.63$	$75.14$	$80.79$	$67.4$	$73.4$	$2.498$
int3	$[8,4,8\rightarrow 2]$	$55.63$	$75.88$	$80.12$	$74.01$	$80.36$	$67.96$	$72.33$	$2.549$
	$[8,4,2,8\rightarrow 2]$	$54.35$	$76.85$	$79.33$	$74.6$	$80.47$	$67.4$	$72.17$	$2.543$
	$[8,4,2,8\rightarrow 4;2]$	$55.2$	$76.98$	$82.45$	$73.59$	$80.41$	$68.43$	$72.84$	$2.58$

🔼 This table displays the results of downstream evaluations and perplexity calculations for the Matryoshka Quantization (MatQuant) model using OmniQuant on the Gemma-2 9B model. It shows the impact of co-distillation on the model’s performance across different precision levels (int8, int4, int2, int6, int3). The table presents the average accuracy across various downstream tasks and the log perplexity scores on a validation set. Different co-distillation configurations are compared to assess their effects on downstream performance and model complexity. Each row corresponds to a specific set of hyperparameters used during the training process.
read the caption
Table 16: Table presents the downstream evaluation and perplexity results for our MatQuantMatQuant{\rm MatQuant}roman_MatQuant co-distillation experiments on Gemma-2 9B with OmniQuant.

Data type	Config.	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average	log pplx.
	QAT	Gemma-2 9B
int8	$[8,4,8\rightarrow 2]$	$58.11$	$76.43$	$81.25$	$79.12$	$82.05$	$71.35$	$74.72$	$2.298$
	$[8,4,2,8\rightarrow 2]$	$57.51$	$76.43$	$81.53$	$78.95$	$82.1$	$71.19$	$74.62$	$2.299$
	$[8,4,2,8\rightarrow 4;2]$	$58.11$	$76.14$	$81.68$	$79.12$	$82.26$	$71.51$	$74.8$	$2.302$
int4	$[8,4,8\rightarrow 2]$	$57.42$	$76.35$	$77.55$	$78.06$	$81.61$	$71.59$	$73.76$	$2.328$
	$[8,4,2,8\rightarrow 2]$	$56.91$	$75.8$	$78.44$	$77.76$	$81.39$	$72.38$	$73.78$	$2.329$
	$[8,4,2,8\rightarrow 4;2]$	$57.51$	$75.76$	$75.96$	$77.96$	$81.72$	$71.98$	$73.48$	$2.33$
int2	$[8,4,8\rightarrow 2]$	$39.51$	$65.03$	$66.88$	$63.37$	$75.08$	$61.01$	$61.81$	$2.74$
	$[8,4,2,8\rightarrow 2]$	$40.78$	$66.5$	$67.55$	$63.67$	$75.95$	$60.62$	$62.51$	$2.746$
	$[8,4,2,8\rightarrow 4;2]$	$40.19$	$65.7$	$65.57$	$63.83$	$75.3$	$62.12$	$62.12$	$2.746$
int6	$[8,4,8\rightarrow 2]$	$57.85$	$76.09$	$81.47$	$78.98$	$81.88$	$71.27$	$74.59$	$2.301$
	$[8,4,2,8\rightarrow 2]$	$57.17$	$75.97$	$82.2$	$79$	$81.83$	$71.9$	$74.68$	$2.302$
	$[8,4,2,8\rightarrow 4;2]$	$57.42$	$76.09$	$82.29$	$78.95$	$82.10$	$71.27$	$74.69$	$2.305$
int3	$[8,4,8\rightarrow 2]$	$51.96$	$71.55$	$78.07$	$73.17$	$79.43$	$66.93$	$70.18$	$2.485$
	$[8,4,2,8\rightarrow 2]$	$50.94$	$71.76$	$78.78$	$73.09$	$79.05$	$66.77$	$70.06$	$2.486$
	$[8,4,2,8\rightarrow 4;2]$	$51.45$	$72.39$	$78.84$	$73.46$	$79.6$	$67.96$	$70.62$	$2.731$

🔼 Table 17 presents the downstream evaluation and perplexity results for the Matryoshka Quantization (MatQuant) co-distillation experiments on the 9B parameter Gemma-2 model using Quantization Aware Training (QAT). It shows the performance across various downstream tasks (ARC-c, ARC-e, BoolQ, HellaSwag, PIQA, Winogrande) and the associated perplexity scores on a validation set. Different configurations of MatQuant co-distillation are compared, varying the precision levels (int8, int4, int2, int6, int3) and whether or not co-distillation is applied. The table allows readers to compare the effects of different co-distillation strategies on downstream task performance and perplexity.
read the caption
Table 17: Table presents the downstream evaluation and perplexity results for our MatQuantMatQuant{\rm MatQuant}roman_MatQuant co-distillation experiments on Gemma-2 9B with QAT.

Data type	Method	Gemma-2 9B
		ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$58.96$	$77.57$	$83.33$	$77.31$	$81.12$	$67.96$	$74.38$
int8	Baseline	$58.62$	$77.02$	$83.43$	$79.01$	$81.34$	$68.27$	$74.61$
int8	${\rm MatQuant}$	$59.04$	$77.9$	$84.4$	$78.76$	$81.12$	$69.22$	$75.07$
int4	Sliced int8	$57.42$	$76.73$	$81.62$	$76.02$	$80.58$	$68.98$	$73.56$
	Baseline	$56.06$	$74.96$	$79.27$	$77.83$	$80.25$	$69.53$	$72.98$
	${\rm MatQuant}$	$57.34$	$76.77$	$84.19$	$77.51$	$80.74$	$68.11$	$74.11$
int2	Sliced int8	$24.74$	$25.63$	$58.53$	$25.5$	$50.71$	$49.17$	$39.05$
	Baseline	-	-	-	-	-	-	-
	S.P. ${\rm MatQuant}$	$24.91$	$41.62$	$62.26$	$40.87$	$63.38$	$53.67$	$47.78$
	${\rm MatQuant}$	$28.24$	$39.23$	$62.17$	$39.13$	$63.49$	$50.75$	$47.17$
int6	Sliced int8	$58.53$	$77.15$	$82.48$	$79.04$	$81.5$	$68.67$	$74.56$
	Baseline	$58.87$	$77.06$	$83.12$	$78.81$	$81.23$	$68.82$	$74.65$
	${\rm MatQuant}$	$59.81$	$77.9$	$84.8$	$78.68$	$81.07$	$67.96$	$75.04$
int3	Sliced int8	$43.6$	$64.98$	$72.66$	$66$	$75.95$	$62.19$	$64.23$
	Baseline	-	-	-	-	-	-	-
	S.P. ${\rm MatQuant}$	$50.85$	$73.11$	$71.13$	$72.01$	$79.38$	$65.67$	$68.69$
	${\rm MatQuant}$	$45.22$	$69.32$	$78.5$	$68.72$	$76.01$	$63.85$	$66.94$

🔼 This table presents the results of a downstream evaluation on the Gemma-2 9B model. The evaluation assesses the performance of Matryoshka Quantization (MatQuant) when applied to Feed-Forward Network (FFN) and Attention layers using Quantization Aware Training (QAT). The table compares MatQuant’s performance across different quantization levels (int8, int4, int2, int6, and int3), showing accuracy and perplexity scores. It also includes results for baselines (models trained without MatQuant) and sliced models (models created by directly truncating higher precision weights). The purpose of the table is to demonstrate the effectiveness and flexibility of MatQuant in achieving high accuracy at various bit-widths, especially lower precisions like int2.
read the caption
Table 18: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant FFN + Attention quantization on Gemma-2 9B with QAT.

Data type	Method	Mistral 7B
		ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average
bfloat16		$49.57$	$73.74$	$84.4$	$80.61$	$81.18$	$74.43$	$73.99$
int8	Baseline	$49.23$	$72.9$	$83.49$	$80.26$	$81.28$	$75.22$	$73.73$
int8	${\rm MatQuant}$	$49.32$	$72.31$	$83.76$	$80.2$	$81.18$	$74.74$	$73.58$
int4	Sliced int8	$45.99$	$71.76$	$81.41$	$76.95$	$80.41$	$71.98$	$71.42$
	Baseline	$48.04$	$71.72$	$78.87$	$78.93$	$80.36$	$73.32$	$71.87$
	${\rm MatQuant}$	$47.01$	$69.95$	$82.02$	$76.81$	$80.25$	$72.93$	$71.5$
int2	Sliced int8	$22.78$	$24.03$	$58.75$	$24.63$	$50.54$	$49.64$	$38.39$
	Baseline	-	-	-	-	-	-	-
	S.P. ${\rm MatQuant}$	$23.21$	$23.82$	$37.83$	$24.67$	$49.02$	$49.57$	$34.69$
	${\rm MatQuant}$	$22.27$	$32.49$	$62.02$	$32.43$	$59.3$	$51.46$	$43.33$
int6	Sliced int8	$49.32$	$73.53$	$82.66$	$80.16$	$81.12$	$75.45$	$73.71$
	Baseline	$49.32$	$73.4$	$82.48$	$80.24$	$81.28$	$75.61$	$73.72$
	${\rm MatQuant}$	$49.15$	$71.76$	$83.73$	$80.13$	$81.18$	$74.19$	$73.36$
int3	Sliced int8	$20.65$	$31.57$	$44.34$	$28.79$	$59.41$	$51.38$	$39.36$
	Baseline	-	-	-	-	-	-	-
	S.P. ${\rm MatQuant}$	$41.98$	$65.53$	$79.39$	$74.42$	$79.22$	$69.93$	$68.41$
	${\rm MatQuant}$	$34.64$	$55.13$	$70.43$	$58.61$	$73.39$	$64.48$	$59.45$

🔼 Table 19 presents the downstream evaluation results for the Matryoshka Quantization (MatQuant) technique applied to the Feed-Forward Network (FFN) and Attention layers of the Mistral 7B model using Quantization Aware Training (QAT). It shows the performance (accuracy on downstream tasks and log perplexity on a validation set) of different quantization levels (int8, int4, int2, int6, int3) achieved using MatQuant, as compared to baselines where models are trained separately for each level. It also includes results for models obtained via interpolation from an int8 model using MatQuant’s multi-scale capability. This table illustrates MatQuant’s effectiveness in training a single model capable of performing well at various precision levels, and its ability to generate high-quality interpolated models.
read the caption
Table 19: Table presents the downstream evaluation results for MatQuantMatQuant{\rm MatQuant}roman_MatQuant FFN + Attention quantization on Mistral 7B with QAT.

	int2	Gemma2-2B
	Method	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Task Avg.	$\log$ pplx.
OmniQuant	S.P. ${\rm MatQuant}$	$34.64$	$64.06$	$65.69$	$53.07$	$69.7$	$57.14$	$57.38$	$3.185$
	Baseline	$31.31$	$53.58$	$62.2$	$40.78$	$66.05$	$54.06$	$51.33$	$3.835$
	${\rm MatQuant}$	$34.39$	$59.64$	$62.69$	$52.11$	$69.86$	$55.56$	$55.71$	$3.292$
QAT	S.P. ${\rm MatQuant}$	$28.92$	$53.79$	$62.84$	$48.41$	$69.86$	$55.25$	$53.18$	$3.090$
	Baseline	$24.66$	$43.22$	$62.17$	$38.39$	$64.42$	$53.59$	$47.74$	$3.433$
	${\rm MatQuant}$	$28.24$	$51.73$	$64.19$	$46.76$	$68.66$	$55.01$	$52.43$	$3.153$

🔼 Table 20 presents a comparison of the performance of three different quantization methods: Single Precision MatQuant, MatQuant, and a baseline method. All three methods are applied to a Gemma-2 2B model, using both OmniQuant and QAT (two different quantization training approaches). The table shows downstream evaluation metrics (accuracy on various tasks) and perplexity scores for int2 quantization, highlighting the trade-offs in accuracy and computational cost among the different techniques.
read the caption
Table 20: Table presents downstream evaluation and perplexity results for Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant, comparing it with MatQuantMatQuant{\rm MatQuant}roman_MatQuant and the Baseline for int2 quatization of Gemma-2 2B with OmniQuant and QAT.

		Gemma-2 9B
Data type	Method	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average	$\log$ pplx.
int8	S.P. ${\rm MatQuant}$	$56.48$	$76.85$	$73.36$	$74.87$	$80.74$	$66.77$	$71.51$	$2.525$
	OmniQuant	$59.47$	$77.31$	$83.94$	$77.35$	$81.39$	$68.11$	$74.59$	$2.418$
	MatQuant	$58.11$	$78.03$	$83.27$	$76.17$	$81.18$	$67.09$	$73.97$	$2.451$
int4	S.P. ${\rm MatQuant}$	$57.17$	$77.02$	$74.28$	$74.41$	$80.69$	$67.56$	$71.85$	$2.543$
	OmniQuant	$58.79$	$78.37$	$83.55$	$76.71$	$81.45$	$67.09$	$74.33$	$2.451$
	MatQuant	$57.25$	$77.36$	$84.86$	$75.52$	$81.5$	$66.77$	$73.88$	$2.481$
int2	S.P. ${\rm MatQuant}$	$49.74$	$74.66$	$80.92$	$66.57$	$76.06$	$63.54$	$68.58$	$2.857$
	OmniQuant	$39.16$	$63.43$	$72.11$	$52.24$	$72.63$	$61.88$	$60.24$	$3.292$
	MatQuant	$48.72$	$72.18$	$79.2$	$68.11$	$76.17$	$66.77$	$68.52$	$2.809$

🔼 This table presents a comparison of downstream evaluation and perplexity results for three different quantization methods applied to the Gemma-2 9B model: Single Precision MatQuant, MatQuant, and a baseline. Single Precision MatQuant is a variation where the model is specifically trained only for 2-bit precision (int2), and the int4 and int8 models are derived by slicing the weights from this trained model. MatQuant, in contrast, jointly optimizes for multiple bit-widths (int2, int4, int8). The baseline represents models trained independently for each bit-width. The comparison allows assessing the performance of Single Precision MatQuant relative to MatQuant and the standard method of independent training for each precision. The results are provided for OmniQuant.
read the caption
Table 21: Table presents downstream evaluation and perplexity results for Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant, comparing it with MatQuantMatQuant{\rm MatQuant}roman_MatQuant and the Baseline for int2, int4, int8 quatization of Gemma-2 9B with OmniQuant. Note that the model was trained with Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant for int2, the int4 and int8 model were sliced post training.

		Gemma-2 9B
Data type	Method	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Average	log pplx.
int8	S.P. ${\rm MatQuant}$	$55.97$	$76.18$	$80.09$	$75.43$	$80.69$	$68.9$	$72.88$	$2.429$
	QAT	$47.78$	$70.66$	$75.08$	$69.92$	$78.35$	$65.11$	$67.82$	$2.29$
	MatQuant	$46.25$	$71.21$	$75.6$	$69.97$	$78.4$	$64.64$	$67.68$	$2.301$
int4	S.P. ${\rm MatQuant}$	$55.2$	$76.01$	$74.74$	$74.19$	$80.41$	$68.9$	$71.57$	$2.429$
	QAT	$46.16$	$71.59$	$73.73$	$68.72$	$78.62$	$63.38$	$67.03$	$2.324$
	MatQuant	$44.37$	$70.45$	$75.81$	$68.43$	$78.35$	$64.88$	$67.05$	$2.332$
int2	S.P. ${\rm MatQuant}$	$41.21$	$66.2$	$65.02$	$64.31$	$76.06$	$62.35$	$62.53$	$2.706$
	QAT	$33.45$	$55.43$	$62.26$	$54.8$	$70.51$	$59.67$	$56.02$	$2.923$
	MatQuant	$39.85$	$65.66$	$65.93$	$64.08$	$75.68$	$62.75$	$62.32$	$2.756$

🔼 Table 22 presents a comparison of downstream evaluation and perplexity results for three different quantization methods applied to the Gemma-2 9B model using QAT. The methods compared are: the baseline (explicitly trained models for each precision), MatQuant (a multi-scale quantization method), and Single Precision MatQuant (a variation of MatQuant focusing solely on int2 precision). Crucially, Single Precision MatQuant only trained for int2 precision, while the int4 and int8 models were derived by slicing the trained weights, highlighting its ability to generate multiple precision models from a single training process. The table details the performance of each method across different precision levels (int2, int4, int8), showing accuracy and perplexity scores for each.
read the caption
Table 22: Table presents downstream evaluation and perplexity results for Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant, comparing it with MatQuantMatQuant{\rm MatQuant}roman_MatQuant and the Baseline for int2, int4, int8 quatization of Gemma-2 9B with QAT. Note that the model was trained with Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant for int2, the int4 and int8 model were sliced post training.

	int2	Mistral 7B
	Method	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	Winogrande	Task Avg.	$\log$ pplx.
OmniQuant	S.P. ${\rm MatQuant}$	$39.93$	$66.25$	$76.97$	$72.99$	$78.07$	$69.93$	$67.36$	$2.464$
	Baseline	$36.69$	$61.36$	$70.06$	$57.47$	$70.67$	$62.19$	$59.74$	$3.931$
	${\rm MatQuant}$	$41.38$	$67.42$	$71.62$	$71.98$	$77.86$	$65.67$	$65.99$	$2.569$
QAT	S.P. ${\rm MatQuant}$	$34.64$	$56.19$	$70.73$	$66.77$	$75.52$	$65.43$	$61.55$	$2.435$
	Baseline	$29.78$	$48.23$	$64.5$	$55.11$	$70.84$	$61.25$	$54.95$	$2.694$
	${\rm MatQuant}$	$34.3$	$55.09$	$71.83$	$65.89$	$75.52$	$65.11$	$61.29$	$2.474$

🔼 Table 23 presents a comparison of the downstream evaluation and perplexity results for three different methods of quantizing the Mistral 7B model to 2 bits: Single Precision MatQuant, MatQuant, and a baseline method. Both OmniQuant and QAT training methods are included in the comparison. The table shows the results across several evaluation metrics, providing a comprehensive analysis of each method’s performance.
read the caption
Table 23: Table presents downstream evaluation and perplexity results for Single⁢ ⁢Precison⁢ ⁢MatQuantSingle Precison MatQuant{\rm Single\text{ }Precison\text{ }MatQuant}roman_Single roman_Precison roman_MatQuant, comparing it with MatQuantMatQuant{\rm MatQuant}roman_MatQuant and the Baseline for int2 quatization of Mistral 7B with OmniQuant and QAT.

TL;DR#

Key Takeaways#

Why does it matter?#

Visual Insights#

In-depth insights#

MatQuant: Multi-Scale#

Int2 Accuracy Boost#

Mix’n’Match Quantization#

Interpolative Behavior#

Future: FP Extension#

More visual insights#

Full paper#