Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient

🔼 This figure compares image generation results between the original VAR-d30 model and the proposed VAR-CoDe model on the ImageNet 256x256 dataset. The top row shows images generated using the original VAR-d30 model, highlighting its generation time of 3.62 seconds and memory usage of 40GB. The bottom row displays images generated by the VAR-CoDe method, demonstrating a 1.7x speedup (2.11 seconds) and a 50% reduction in memory usage (20GB). Importantly, the visual quality remains almost identical between the two methods, indicating negligible quality degradation due to the efficiency improvements achieved by VAR-CoDe.

read the caption

Figure 2: Comparison of generation results between original VAR-d30 (up) and our VAR-CoDE (bottom) for ImageNet 256×\times×256. Our method achieves 1.7x speedup (3.62s to 2.11s), and needs only 0.5x memory space (40GB to 20GB), with negligible quality degradation.

🔼 Figure 3 demonstrates three key aspects of the Collaborative Decoding (CoDe) method. (a) It shows that increasing model parameters is more effective in improving the quality of generated content at smaller scales than at larger scales. This is done by using different sized models to generate tokens at different scales. (b) Fourier analysis shows that smaller scales generate low-frequency information, while larger scales generate high-frequency detail. (c) Shows the effect of varying the number of drafting and refining tokens on the final model’s performance.

read the caption

Figure 3: (a) Effectiveness of increasing parameters at the k𝑘kitalic_k-th scale is evaluated by predicting token map rksubscript𝑟𝑘r_{k}italic_r start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT using four VAR models with different parameter sizes (2B, 1B, 0.6B, and 0.3B), while other scales (r1,r2,…,rk−1,rk+1,…,r10)subscript𝑟1subscript𝑟2…subscript𝑟𝑘1subscript𝑟𝑘1…subscript𝑟10(r_{1},r_{2},\dots,r_{k-1},r_{k+1},\dots,r_{10})( italic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_r start_POSTSUBSCRIPT italic_k - 1 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT italic_k + 1 end_POSTSUBSCRIPT , … , italic_r start_POSTSUBSCRIPT 10 end_POSTSUBSCRIPT ) are generated using the largest VAR-d30 model. (b) Fourier spectrum analysis is conducted on generated content at the first 3 scales and the last 3 scales. (c) Training-free performance comparison of model collaboration decoding across various settings of draft tokens M𝑀Mitalic_M and refiner tokens 680−M680𝑀680-M680 - italic_M.

🔼 This figure illustrates the collaborative decoding process used in the proposed CoDe method. A large VAR model (the drafter) initially generates low-frequency token maps for smaller scales (up to scale N, where N=6 in this example). Then, a smaller VAR model (the refiner), using these token maps as a prefix, efficiently predicts the remaining high-frequency token maps for larger scales. Both models undergo fine-tuning using ground truth labels and teacher logits to optimize their performance at specific scales.

read the caption

Figure 4: Overview of the collaborative decoding process, we use a drafting step N=6𝑁6N=6italic_N = 6 for instance. CoDe uses a large VAR model as the drafter ϵθdsubscriptitalic-ϵsubscript𝜃𝑑\epsilon_{\theta_{d}}italic_ϵ start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT to generate the token maps RL=(r1,r2,…,rN)subscript𝑅𝐿subscript𝑟1subscript𝑟2…subscript𝑟𝑁R_{L}=(r_{1},r_{2},\ldots,r_{N})italic_R start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT = ( italic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_r start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ) at smaller scales. The small refiner model ϵθrsubscriptitalic-ϵsubscript𝜃𝑟\epsilon_{\theta_{r}}italic_ϵ start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT italic_r end_POSTSUBSCRIPT end_POSTSUBSCRIPT then uses RLsubscript𝑅𝐿R_{L}italic_R start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT as an initial prefix to efficiently predict the remaining token maps RH=(rN+1,rN+2,…,rK)subscript𝑅𝐻subscript𝑟𝑁1subscript𝑟𝑁2…subscript𝑟𝐾R_{H}=(r_{N+1},r_{N+2},\ldots,r_{K})italic_R start_POSTSUBSCRIPT italic_H end_POSTSUBSCRIPT = ( italic_r start_POSTSUBSCRIPT italic_N + 1 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT italic_N + 2 end_POSTSUBSCRIPT , … , italic_r start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ) at larger scales. Both models are fine-tuned on their designated predictive scales using ground truth labels rk∗superscriptsubscript𝑟𝑘r_{k}^{*}italic_r start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT and teacher logits pteacher⁢(rk)subscript𝑝teachersubscript𝑟𝑘p_{\text{teacher}}(r_{k})italic_p start_POSTSUBSCRIPT teacher end_POSTSUBSCRIPT ( italic_r start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ), respectively.

🔼 Figure 5 presents a comparison of different image generation methods, focusing on efficiency and quality. (a) shows that CoDe achieves the best balance between speed and FID score compared to other methods. (b) compares the inference latency of CoDe and VAR-d30 across different batch sizes, demonstrating CoDe’s efficiency. Finally, (c) shows that in the multi-scale decoding process of CoDe, the refiner model is more efficient at larger scales than the drafter model.

read the caption

Figure 5: (a) Our CoDe demonstrates the optimal efficiency-quality trade-off among all evaluated methods. (b) Inference latency is measured across varying batch sizes for the original VAR-d30, our CoDe (N=6), and the VQVAE decoder. (c) We analyze the time cost associated with parallel decoding at each scale, showing that the refiner model is significantly more efficient than the drafter at larger scales.

🔼 This figure displays a qualitative comparison of image generation results between the original VAR-d30 model and the proposed CoDe model with varying drafting steps (N=6, 7, 8). Each row shows generated images from a model, highlighting the speed and memory efficiency gains achieved by CoDe while maintaining comparable image quality. The original VAR-d30 serves as the baseline, illustrating the trade-off between speed and quality as the number of drafting steps decreases.

read the caption

Figure 6: Qualitative comparison between the original VAR-d30 model and our proposed CoDe model, with different drafting steps.

🔼 This figure showcases the zero-shot generalization capabilities of the Collaborative Decoding (CoDe) method on image inpainting and image editing tasks. The results demonstrate CoDe’s ability to perform these tasks without any additional training. The images presented illustrate examples of the inpainting and editing results achieved using CoDe.

read the caption

Figure 7: Qualitative results of CoDe’s zero-shot generalization on image inpainting and image editing.

🔼 This figure shows a comparison of image generation results between the original VAR-d16 model and a version fine-tuned using a perturbation technique. The ‘Up’ section displays images generated by the standard VAR-d16 model, showcasing its typical output. The ‘Down’ section shows images generated after the model underwent perturbation fine-tuning, which selectively modifies the model’s parameters for higher-frequency components in the final scales, specifically focusing on details. By comparing the two sets of images, one can visually assess the impact of this perturbation method on the generated image quality and detail preservation.

read the caption

Figure 8: Up: images generated by the original VAR-d16 models. Down: images generated by the perturbation fine-tuned VAR-d16.

🔼 This figure presents a qualitative comparison of images generated by the original VAR-d30 model and the CoDe model with varying numbers of drafting steps (N=9, 8, 7, 6). Each row shows a set of generated images for the same set of prompts. The goal is to visually demonstrate how CoDe’s speed improvements (shown numerically in the caption) do not significantly impact image quality. The increasing speed is achieved by reducing the number of drafting steps and thereby reducing computational demands. A visual inspection allows assessing whether the speed improvements come at the cost of image quality.

read the caption

Figure 9: Qualitative comparison between the original VAR-d30 model and our proposed CoDe model, with different drafting steps.

🔼 Figure 10 presents a qualitative comparison of image generation results between the original VAR-d30 model and the proposed CoDe model with varying numbers of drafting steps (N). Each row displays images generated using a different model (original VAR-d30 or CoDe with N=9, 8, 7, or 6). The figure visually demonstrates the impact of reducing the number of drafting steps on image quality while increasing speed and reducing memory usage. The FID (Fréchet Inception Distance) scores for each model are also provided, showing a small increase in FID (a measure of image quality) as the number of drafting steps decreases, and speedup factors are given.

read the caption

Figure 10: Qualitative comparison between the original VAR-d30 model and our proposed CoDe model, with different drafting steps.

🔼 This table shows the impact of specialized fine-tuning on the Collaborative Decoding (CoDe) method. It presents FID scores and speedup factors for different numbers of drafting steps (N) in the CoDe framework. The ‘Training-free’ row indicates the performance without any fine-tuning, showing the performance baseline. The other rows illustrate the improvements achieved through fine-tuning the model. This allows for a comparison of the performance trade-offs between training-free operation and the benefits of specialized fine-tuning.

read the caption

Table 2: Effect of specialized fine-tuning

🔼 This table compares the memory usage of the original VAR model and the proposed CoDe model across different batch sizes (8, 16, 32, 64, and 128). For each batch size, it breaks down the memory consumption into three categories: running memory (memory used during the model’s execution), KV cache (memory used for storing key-value pairs in the attention mechanism), and model parameters. The total memory usage is also shown. The table demonstrates how CoDe significantly reduces memory usage compared to the original VAR model, especially for larger batch sizes. Out of memory (OOM) errors occur for the original VAR at batch size 128, which are avoided by CoDe.

read the caption

Table 3: Memory usage comparison across different batch sizes

🔼 This table shows the impact of increasing model parameters at different scales on image generation quality. The experiment predicts tokens for a single scale using models of varying sizes (0.3B, 0.6B, 1.0B, and 2.0B parameters), while using the largest model (2B) for all other scales. The results demonstrate that increasing parameters significantly improves quality at smaller scales, but this improvement diminishes as the scale increases; at the final scale, the largest model shows little advantage over the smallest.

read the caption

Table 4: Impact of increasing parameters across scales

🔼 This table presents a comparison of image quality metrics for different models. It assesses the performance of the original VAR-d30 model and several variants of CoDe (Collaborative Decoding) with varying numbers of drafting steps (N). The metrics used are not reference-based (meaning they don’t compare to a ground truth image), instead relying on metrics that evaluate the image quality itself. The metrics included are MUSIQ, CLIPIQA, and NIQE, each offering a different perspective on image quality, providing a more comprehensive assessment than FID and IS alone. The table shows how different models trade off efficiency (speedup, throughput) with various image quality aspects.

read the caption

Table 5: No reference metrics for additional image quality assessments.

🔼 This table presents the performance of the Collaborative Decoding (CoDe) method without any additional training. It shows the FID (Fréchet Inception Distance), Inception Score (IS), Precision, and Recall metrics for different numbers of drafting steps (N) in CoDe. Lower FID scores indicate better image quality. Higher IS, precision, and recall values suggest improved image generation performance.

read the caption

Table 6: The training-free performance of CoDe