Autoregressive Image Generation with Randomized Parallel Decoding

🔼 Figure 2 illustrates the differences in how three different autoregressive models (RAR, RandAR, and ARPG) handle the positional information of the next predicted token during image generation. RAR fuses positional information into the current token using additive positional embedding. RandAR explicitly inserts the position as an instructional token in the input sequence. ARPG integrates positional information into the query of the attention mechanism, decoupling positional guidance from content representation. The corresponding probability models for RAR and RandAR are shown, highlighting their differences in incorporating position information.

read the caption

Figure 2: Comparison of different methods. RAR [50]: The position of the next predicted token is fused into the current token via additive positional embedding. Its joint probability distribution is defined as ∏i=1np⁢(xτi∣xτ1+pτ2,…,xτi−1+pτi)superscriptsubscriptproduct𝑖1𝑛𝑝conditionalsubscript𝑥subscript𝜏𝑖subscript𝑥subscript𝜏1subscript𝑝subscript𝜏2…subscript𝑥subscript𝜏𝑖1subscript𝑝subscript𝜏𝑖\prod_{i=1}^{n}p\big{(}x_{\tau_{i}}\mid x_{\tau_{1}}+p_{\tau_{2}},\dots,x_{% \tau_{i-1}}+p_{\tau_{i}}\big{)}∏ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_p ( italic_x start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∣ italic_x start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT + italic_p start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT , … , italic_x start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT + italic_p start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ). RandAR [25]: The position of the next predicted token is explicitly inserted as an instructional token within the input sequence. This yields the probability distribution ∏i=1np⁢(xτi∣xτ1,pτ2,…,xτi−1,pτi)superscriptsubscriptproduct𝑖1𝑛𝑝conditionalsubscript𝑥subscript𝜏𝑖subscript𝑥subscript𝜏1subscript𝑝subscript𝜏2…subscript𝑥subscript𝜏𝑖1subscript𝑝subscript𝜏𝑖\prod_{i=1}^{n}p\big{(}x_{\tau_{i}}\mid x_{\tau_{1}},p_{\tau_{2}},\dots,x_{% \tau_{i-1}},p_{\tau_{i}}\big{)}∏ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_p ( italic_x start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∣ italic_x start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT , italic_p start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT , … , italic_x start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT , italic_p start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT ). ARPG (ours): The position of the next predicted token is integrated as a query within the attention mechanism. The corresponding probabilistic model, formalized in Eq. (8).

🔼 This figure shows the architecture of the ARPG model, a two-pass decoder architecture. The first pass uses self-attention layers to extract contextual representations of image tokens as key-value pairs. The second pass uses cross-attention layers with target-aware queries that attend to these key-value pairs to guide the prediction. During training, the number of queries matches the number of key-value pairs. The positional embedding of each key reflects its actual position, while the positional embedding of each query is shifted right to align with its target position. During inference, multiple queries are input simultaneously, sharing a common KV cache to enable parallel decoding. This architecture enables training and inference in fully random token order.

read the caption

(a) Architecture

🔼 This figure illustrates the training and inference mechanisms of the guided decoding process in ARPG. During training, the number of queries matches the number of key-value pairs. Each query’s positional embedding is right-shifted to align with its target position, while each key’s positional embedding reflects its actual position. This allows the model to learn to predict tokens at random positions. During inference, however, multiple queries are input concurrently, sharing a common KV cache to enable parallel decoding. This shared cache drastically improves efficiency.

read the caption

(b) Training and inference details of guided-decoding

🔼 Figure 3 illustrates the ARPG architecture, a two-pass decoder model. The first pass uses self-attention layers to process the image token sequence and generate global key-value pairs representing contextual information. The second pass employs cross-attention layers with ’target-aware queries’ to focus on specific parts of the global key-value pairs, guiding the prediction process. During training, the number of queries equals the number of key-value pairs. Each key’s positional embedding indicates its position in the sequence, while each query’s embedding is offset to point to the position of the token to be predicted. This design allows for parallel decoding during inference because multiple queries can be processed simultaneously using a shared key-value cache.

read the caption

Figure 3: 3(a) ARPG architecture: We employ a Two-Pass Decoder architecture. In the first pass, N𝑁Nitalic_N self-attention layers extract contextual representations of the image token sequence as global key-value pairs. In the second pass, N𝑁Nitalic_N cross-attention layers use target-aware queries that attend to these global key-value pairs to guide prediction. 3(b) During training, the number of queries matches the number of key-value pairs. Each key’s positional embedding reflects its actual position, while each query’s positional embedding is right-shifted to align with its target position. During inference, multiple queries are input simultaneously, sharing a common KV cache to enable parallel decoding.

🔼 This figure shows several images generated by the ARPG model given different class labels. It demonstrates the model’s ability to generate high-quality, diverse images conditioned on class information. Each image in the grid represents a different class.

read the caption

(a) Class-Conditional Image Generation

🔼 This figure shows example images generated by the ARPG model under the control of additional information, such as canny edges and depth maps. The model is able to generate images that incorporate these additional inputs, demonstrating its capability for controllable image synthesis. This showcases the flexibility of the ARPG approach in creating images beyond simple class-conditional generation.

read the caption

(b) Controllable Image Generation

🔼 Figure 4 presents example images generated by the ARPG model. Subfigure 4(a) showcases class-conditional image generation, where the model generates images based solely on a given class label. Subfigure 4(b) demonstrates controllable image generation, in which the model generates images guided by both a class label and additional conditional information, specifically canny edges and depth maps. This illustrates the model’s ability to incorporate external cues to influence the image synthesis process. All images shown were generated with 64 sampling steps.

read the caption

Figure 4: Generated samples of ARPG on ImageNet-1K 256×\times×256: 4(a) Class-conditional image generation. 4(b) Controllable image generation with canny edges and depth maps as conditions respectively. All images are sampled using 64 steps.

🔼 Figure 5 showcases the model’s zero-shot capabilities on image manipulation tasks without explicit training. The top row demonstrates inpainting (filling missing parts of an image) and class-conditional editing (modifying an existing image based on a class label). The bottom row shows outpainting (extending an image beyond its original boundaries). This highlights the model’s ability to generalize to various tasks beyond the standard image generation.

read the caption

Figure 5: Samples of zero-shot inference. Top: Inpainting and class-conditional editing. Bottom: Outpainting.

🔼 Figure A illustrates the process of controllable image generation using the ARPG model. It focuses on how the model handles the interaction between queries and keys in the attention mechanism, specifically highlighting how these elements guide the generation process. To simplify the illustration, raster-order generation is used as an example, and the ‘value’ component of the attention mechanism is omitted. The class token ([CLS]) is also shown.

read the caption

Figure A: Implementation Details of Controllable Image Generation. For clarity, we illustrate the process using raster-order as an example. The figure only illustrates the interaction between query and key in the attention mechanism and its output, omitting the value for simplicity. [CLS]: Class token.

🔼 Table 2 presents a comprehensive comparison of various autoregressive image generation models on the ImageNet 256x256 dataset. The models are categorized into diffusion, masked, and autoregressive models (both raster and random order). For each model, the table shows the number of parameters, the number of generation steps, the image generation throughput (images per second), memory usage (in GB), Fréchet Inception Distance (FID, lower is better, indicating higher image quality), Inception Score (IS, higher is better, indicating better image diversity), precision, and recall. This allows for a direct comparison of model performance across different approaches, highlighting the relative strengths and weaknesses of each method in terms of speed, memory efficiency, and image quality.

read the caption

Table 2: Overall comparisons on class-conditional ImageNet 256×\times×256 benchmark. The symbols “↓↓\downarrow↓” and “↑↑\uparrow↑” indicate that lower and higher values are better. “-re”: rejection sampling. All models were evaluated with a batch size of 64 and at bfloat16 precision.

🔼 This table presents a comparison of different autoregressive image generation models on the ImageNet 512x512 dataset for class-conditional image generation. The models are evaluated using FID (Fréchet Inception Distance) score, Inception Score (IS), and throughput (images per second). Memory usage (in GB) is also reported. The metrics are used to assess the quality and efficiency of each model. The same evaluation configuration as Table 2 was used. ‘OOM’ indicates that a model ran out of memory during evaluation.

read the caption

Table 3: Model comparisons on class-conditional ImageNet 512×\times×512 benchmark. The evaluation configuration is the same as that in Tab. 2. OOM: Out of Memory.

🔼 This table presents a comparison of different models’ performance on controllable image generation tasks using the ImageNet-1K dataset. It shows the Fréchet Inception Distance (FID) scores, a measure of generated image quality, for various models when generating images conditioned on canny edges and depth maps. The FID score for the ControlVAR model is estimated from its histogram data, as explicitly noted. Lower FID scores indicate better image quality.

read the caption

Table 4: Controllable generation on ImageNet-1K. The FID of ControlVAR [23] is estimated based on its histograms [24].

🔼 This ablation study investigates the impact of various design choices on ARPG-L, a model trained for 150 epochs. The baseline model’s performance is compared against variations including extending training to 400 epochs, removing the shared key-value cache, adjusting the number of guided decoder layers, and removing or solely using guided decoder layers. The table assesses each variation across parameters, layers, random generation capability, parallel decoding capability, number of sampling steps, throughput, memory usage, FID score, and Inception Score (IS) to determine the effects on both efficiency and image generation quality.

read the caption

Table 5: Ablation study. We adopt ARPG-L trained for 150 epochs as the baseline. Here, “Longer Training” refers to a 400-epoch training process. “Randomize” and “Parallelize” mean the ability to generate in a random order and perform parallel decoding, respectively.

🔼 Table 6 presents a quantitative analysis of the trade-off between efficiency and image generation quality for different model configurations. The models were evaluated using an NVIDIA A800-80GB GPU, a batch size of 64, and bfloat16 precision. The table shows how various settings (model size, number of steps, and classifier-free guidance scale (wcfg)) impact the FID (Fréchet Inception Distance) score, throughput (images/second), memory usage (in GB), and Inception Score (IS). The wcfg value represents the scaling factor used to achieve the best FID for each model configuration.

read the caption

Table 6: Quantitative evaluation of efficiency-quality trade-off. Models evaluated on NVIDIA A800-80GB GPU with a batch size of 64 and bfloat16 precision. 𝒘cfgsubscript𝒘cfg\bm{w}_{\text{cfg}}bold_italic_w start_POSTSUBSCRIPT cfg end_POSTSUBSCRIPT: The classifier-free guidance scale for achieving the best FID.

🔼 This table presents the results of an ablation study that explores the impact of different image generation orders on a model pre-trained with a random generation order. The study compares the model’s performance (FID, IS, Precision, Recall) when generating images using various orders, including raster scan, spiral-in, spiral-out, Z-curve, alternate, and the random order used during pre-training. This helps understand how the chosen generation order affects the quality and efficiency of the model’s output.

read the caption

Table 7: Effect of different generation orders tuned on a random order based pre-trained mode.