Identifying Sensitive Weights via Post-quantization Integral

🔼 This table shows how the accuracy of the Taylor expansion approximation for the change in loss function (ΔF) varies with different values of λ (lambda). λ controls how close the interpolated weight w’ is to the original weight w. As λ approaches 0, w’ gets closer to w, making the Taylor expansion more accurate. The table compares the actual change in loss (Actual ΔF) to the values predicted by the first-order and second-order terms of the Taylor expansion for various layers of a 16-layer Llama 3.2 1B model.

read the caption

Table 2: Actual Δ⁢FΔ𝐹\Delta Froman_Δ italic_F with different λ𝜆\lambdaitalic_λ.

🔼 This table presents the results of an experiment designed to evaluate the accuracy of the Post-quantization Integral (PQI) method, a novel sensitivity metric. The experiment quantifies the change in the loss function (ΔF) using PQI with varying numbers of intervals used in the numerical integration process. The goal is to demonstrate how well PQI can predict the actual change in the loss function. The actual ΔF for the dataset is provided for comparison, allowing assessment of PQI’s accuracy with different levels of granularity in the approximation.

read the caption

Table 3: Predicted Δ⁢FΔ𝐹\Delta Froman_Δ italic_F with intervals we split. For reference, the actual Δ⁢F⁢(𝐰)Δ𝐹𝐰\Delta F(\bm{\mathbf{w}})roman_Δ italic_F ( bold_w ) on this dataset should be 0.1024.

🔼 This table presents the element-wise average of the Post-quantization Integral (PQI) sensitivity metric for different layers and sub-layers within the Llama language model. The PQI metric quantifies the sensitivity of each weight to quantization, indicating its importance in maintaining model accuracy. Higher values suggest greater sensitivity and thus a larger impact on the model’s performance if that weight is quantized. The table helps to understand the varying sensitivity across different model components, informing strategies for more effective quantization.

read the caption

Table 4: Element-wise average Δ⁢FP⁢Q⁢IΔsubscript𝐹𝑃𝑄𝐼\Delta F_{PQI}roman_Δ italic_F start_POSTSUBSCRIPT italic_P italic_Q italic_I end_POSTSUBSCRIPT of different layers and sublayers.

🔼 This table shows the relationship between the percentage of weights considered ‘significant’ and their cumulative contribution to the total post-quantization integral (PQI) sensitivity metric. The PQI sensitivity metric quantifies how much each weight’s quantization affects the model’s loss. Higher ΔFPQI values indicate greater sensitivity. The table helps illustrate the impact of focusing on a smaller subset of most sensitive weights in the model’s quantization.

read the caption

Table 5: The proportion of significant weights we choose and how much they can cover in total Δ⁢FP⁢Q⁢IΔsubscript𝐹𝑃𝑄𝐼\Delta F_{PQI}roman_Δ italic_F start_POSTSUBSCRIPT italic_P italic_Q italic_I end_POSTSUBSCRIPT.

🔼 This table presents the results of applying the QTIP (Quantization with Trellises and Incoherence Processing) method to the Llama 3.2 1B Base and Instruct models. It shows the performance metrics achieved at different precision levels (2-bit, 3-bit, 4-bit), using various calibration sets, and with the addition of the ReQuant method. Metrics include perplexity on the WikiText-2 benchmark and the MATH score, indicating performance on mathematical reasoning tasks. Sparsity refers to the proportion of weights that are sparsely stored instead of being fully quantized.

read the caption

Table 6: QTIP results for Llama 3.2 1B Base/Instruct models. The entries share the same meaning as Table 7.

🔼 This table presents the results of evaluating various quantization methods (AWQ, SqueezeLLM, and QTIP) on Llama language models (3.2B and 3B). It shows the WikiText-2 perplexity for base models and the 4-shot MATH (Mathematical Problem Solving) evaluation score for instruction following models. The table compares baseline performance with the results obtained after applying the proposed ReQuant method. ‘Fail’ indicates instances where the model’s output could not be properly parsed due to errors.

read the caption

Table 7: WikiText-2 perplexity for base models and 4-shot MATH evaluation for instruction following models. “Fail” means failure to parse model’s output due to garbled characters.

🔼 This table presents the ablation study results on the WikiText-2 dataset, focusing on the impact of outlier selection and significant weight detach on perplexity. It compares the performance of the proposed ReQuant method against baselines, evaluating different settings for outlier selection and gradual weight detachment. The ‘rand’ row provides a control, where outliers and significant weights are randomly selected, highlighting the effectiveness of the proposed selection strategy.

read the caption

Table 8: Ablation results on WikiText-2 perplexity. The “rand” line indicates that 𝐰osubscript𝐰𝑜\bm{\mathbf{w}}_{o}bold_w start_POSTSUBSCRIPT italic_o end_POSTSUBSCRIPT and 𝐰ssubscript𝐰𝑠\bm{\mathbf{w}}_{s}bold_w start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT are picked out randomly from the weights.

🔼 This table presents the inference speed comparison between different quantization methods with and without the proposed ReQuant technique. It shows the time taken for pre-filling (preparing data), decoding (processing the data), and the total inference time for various model sizes and precision levels. The results are useful for assessing the performance impact of ReQuant on inference speed and for understanding the tradeoff between accuracy and speed.

read the caption

Table 9: Inference speed of Dense-and-Sparse decomposition.

🔼 This table lists the hyperparameters used in the experiments for three different post-training quantization methods: AWQ, SqueezeLLM, and QTIP. For each method, it specifies the calibration dataset used, the sequence length of the calibration data, the number of intervals (N) used for the numerical integration in PQI, and the number of times (n) the dataset was sampled for calculations. It also gives the parameters rs and β used in the ReQuant method, representing the percentage of weights detached and the step size for that detachment respectively.

read the caption

Table 10: Experimental hyperparameters.

🔼 This table shows the estimated GPU time (in hours) required for calculating the Post-quantization Integral (PQI) on an A100 GPU. It details the computation time for different model sizes (Llama 3.2 1B and 3.2 3B) and calibration datasets (WikiText-2 and Tulu 3) with varying numbers of samples (n). The size of the dataset used to calculate PQI significantly impacts the computational cost.

read the caption

Table 11: GPU Hours of doing integral on A100.