Randomized Autoregressive Visual Generation

🔼 Figure 2 illustrates the Randomized Autoregressive (RAR) model, designed for visual generation while maintaining compatibility with language modeling frameworks. The left panel demonstrates the RAR training process: input sequences are randomly permuted with a probability r, initially 1 (fully random) and decreasing linearly to 0 during training. This annealing strategy helps the model learn bidirectional contexts by maximizing the likelihood across various permutation orders, eventually converging to a fixed raster scan. The right panel showcases example images generated by the trained RAR model using the ImageNet dataset.

read the caption

Figure 2: Overview of the proposed Randomized AutoRegressive (RAR) model, which is fully compatible with language modeling frameworks. Left: RAR introduces a randomness annealing training strategy to enhance the model’s ability to learn bidirectional contexts. During training, the input sequence is randomly permuted with a probability r𝑟ritalic_r, which starts at 1 (fully random permutations) and linearly decreases to 0, transitioning the model to a fixed scan order, such as raster scan, by the end of training. Right: Randomly selected images generated by RAR, trained on ImageNet.

🔼 Figure 3 illustrates the concept of target-aware positional embeddings within the Randomized Autoregressive (RAR) model. Panel (a) depicts the training process: images are first tokenized into patches (following the Vision Transformer architecture), each patch receiving an initial positional embedding (blue tokens). The token sequence is then randomly permuted. Crucially, a target-aware positional embedding (green tokens) is added to each token to inform the model which token it should predict next. Panels (b) and (c) showcase the importance of these target-aware embeddings. Panel (b) shows a failure scenario where, without them, two different permuted sequences produce identical predictions because the original positional embeddings alone aren’t sufficient to distinguish the correct prediction in the context of a random permutation. Panel (c) demonstrates that the inclusion of target-aware positional embeddings successfully guides the model toward the correct next-token prediction, even with a randomly permuted input sequence.

read the caption

Figure 3: Illustration of the target-aware positional embedding. Subfigure (a) shows the training process of the proposed Randomized AutoRegressive (RAR) model, along with the target-aware position embedding. Following Vision Transformer [19], images are tokenized into patches with original position embeddings (blue tokens). The token sequence is then randomly permuted, with the target-aware positional embeddings (green tokens) added to guide the model. Subfigures (b) and (c) highlight the importance of the target-aware positional embedding: (b) demonstrates a failure case where both permuted sequences yield identical prediction logits, while (c) shows that the target-aware positional embedding correctly guides the model to predict the next token accurately.

🔼 This figure shows the scaling behavior of the RAR model across different sizes (RAR-B, RAR-L, RAR-XL, RAR-XXL). Subfigure (a) presents the training loss curves for each model variant over training steps. Subfigures (b) and (c) illustrate the FID scores (a metric evaluating image generation quality) with and without classifier-free guidance, respectively. The plots demonstrate how larger models generally achieve lower training losses and better FID scores.

read the caption

(a) training losses

🔼 This figure shows the FID scores achieved by different sized RAR models (RAR-B, RAR-L, RAR-XL, RAR-XXL) without using classifier-free guidance during training. The x-axis represents the training steps, showing the FID score progression over the training process. Different lines represent the FID for each model size. The purpose is to demonstrate the impact of model size on the FID score and assess how well the model generalizes.

read the caption

(b) FID scores w/o classifier-free guidance

🔼 This figure shows the FID (Fréchet Inception Distance) scores achieved by different sized RAR models (RAR-B, RAR-L, RAR-XL, RAR-XXL) when using classifier-free guidance during training. Lower FID scores indicate better image generation quality. The x-axis represents the training steps, showing the progress over the training period. The plot demonstrates the improvement in FID score as model size increases and the effectiveness of classifier-free guidance in enhancing the image generation capabilities of the RAR models.

read the caption

(c) FID scores w/ classifier-free guidance

🔼 This figure analyzes the scaling behavior of the Randomized Autoregressive (RAR) model across different sizes. Subfigure (a) shows that as the model size increases, the training loss decreases, indicating improved model training efficiency. Subfigures (b) and (c) present the Fréchet Inception Distance (FID) scores, a metric for evaluating image quality, with and without classifier-free guidance, respectively. Both subfigures show that larger RAR models consistently achieve lower FID scores, demonstrating that scaling up the model significantly improves the image quality generated.

read the caption

Figure 4: Scaling behavior of RAR models. The scaled-up RAR models demonstrate (a) reduced training losses, and improved FID scores both (b) without and (c) with classifier-free guidance.

🔼 This figure displays example images generated by the RAR model at different scales (RAR-B, RAR-L, RAR-XL, and RAR-XXL). The images demonstrate the model’s ability to generate high-quality images across all model sizes. Notably, as the model size increases, the fidelity and diversity of the generated images improve. This improvement is particularly evident in complex or challenging classes, such as the example of a ‘dogsled’ which contains many fine details and multiple objects.

read the caption

Figure 5: Visualization of samples generated by RAR across various model sizes. RAR generates high-quality visual samples across all model sizes. As model size increases, fidelity and diversity improve, especially in challenging classes (e.g., dogsled).

🔼 This figure visualizes six different scan orders for a 16x16 grid (256 tokens). Each subfigure displays one scan order, showing the order in which tokens are processed. The numbers within each grid represent the index of the token according to that scan order. The scan orders visualized are row-major, spiral in, spiral out, z-curve, subsample, and alternate.

read the caption

(a) row-major

🔼 This subfigure shows one of the six different scan orders tested in the paper for image generation. The spiral scan order starts from the center of the image and spirals outwards, processing pixels in a circular pattern. The numbers in the image indicate the sequence in which each token (representing a pixel or a patch of pixels) is processed. This visualization helps illustrate how different scan orders affect the order of information received by the autoregressive model during training and generation.

read the caption

(b) spiral in

🔼 This figure is a visualization of one of six different scan orders used for processing a 16x16 image (256 tokens) within an autoregressive model. Specifically, it showcases the ‘spiral out’ scan order, where the tokens are processed in a spiral pattern, starting from the center and expanding outwards. The numbers in each cell represent the order in which the tokens are processed.

read the caption

(c) spiral out

🔼 This subfigure shows a visualization of the ‘z-curve’ scan order for a 16x16 grid (256 tokens). A z-curve is a space-filling curve that traverses a grid in a pattern resembling the letter ‘Z’. This particular visualization displays the order in which the tokens are processed, with each number representing the index of the token in the scan order.

read the caption

(d) z-curve

🔼 This image shows a visualization of the ‘subsample’ scan order for a 16x16 grid (256 tokens). The numbers represent the order in which the tokens are processed. Unlike a raster scan which would process tokens sequentially, row by row, this subsampling pattern skips tokens in a specific way. The pattern is designed to demonstrate an alternative autoregressive factorization of the image data, which is explored in the paper as a method to improve context modeling.

read the caption

(e) subsample

🔼 This figure visualizes one of the six different scan orders evaluated in the paper for autoregressive image generation. The alternate scan order processes the image tokens in an alternating pattern across rows, starting from the top left, then moving to the second row from the left, and so on. The numbers represent the order in which the tokens are scanned.

read the caption

(f) alternate

🔼 Figure 6 visualizes six different ways of scanning a 16x16 grid (256 tokens), representing different orders for processing image data in an autoregressive model. Each scan order is displayed as a grid where the numbers indicate the order in which the model processes the tokens. This illustrates the impact of different scan orders on how the model learns and generates images, particularly focusing on the tradeoff between unidirectional (raster scan) and bidirectional (randomized scan) processing of the image. The visualization is directly relevant to the exploration of how the model’s ability to learn and utilize bidirectional context is affected by different factorization orders of the image data during training. The figure is important to show the impact on model learning as the various scanning approaches in the ablation study can significantly impact the model’s learning of contextual information in the model.

read the caption

Figure 6: Different scan orders for a 16×16161616\times 1616 × 16 grid (256 tokens). The number indicates the token’s indices in the scanning order.

🔼 Figure 7 showcases a diverse set of images generated by the Randomized Autoregressive (RAR) model. The images demonstrate the model’s ability to generate high-quality, detailed, and visually diverse samples across a wide range of classes and object characteristics, highlighting its strong performance in image generation.

read the caption

Figure 7: Visualization samples from RAR. RAR is capable of generating high-fidelity image samples with great diversity.

🔼 This table presents an ablation study on the randomness annealing strategy used in the RAR model. It shows the impact of varying the start and end epochs of the annealing process on the model’s performance, as measured by FID, IS, Precision, and Recall. The total number of training epochs is fixed at 400. The first row represents training with a purely raster scan order, while the last row shows results from training with purely random scan orders. The gray row indicates the chosen configuration used in the rest of the paper. The table also highlights the importance of the gradual transition between purely random to raster order in the annealing process.

read the caption

Table 2: Different start and end epochs for randomness annealing, with a total of 400 training epochs and model size RAR-L. The final setting is labeled in gray. †: When start epoch and end epoch are both 00 (1st row), the training reverts to a standard raster order training. ‡: When start epoch and end epoch are both 400400400400 (last row), the training becomes a purely random order training. After training is finished, all results are obtained with raster order sampling, except for the purely random order training (i.e., last row), where we also randomly sample the scan order following [36], which otherwise could not produce a reasonable result.

🔼 This table investigates the impact of different image scanning orders on the performance of the RAR-L model. Six common scan orders, including the standard row-major order, are compared. The results show the final FID, Inception Score (IS), precision, and recall after training with each scan order. The default settings used in the experiments are highlighted in gray for easy reference. A visual representation of each scan order is provided in the appendix for better understanding.

read the caption

Table 3: Effect of different scan orders RAR-L converges to. We mainly consider 6 different scan orders (row major, spiral in, spiral out, z-curve, subsample, alternate) as studied in [22]. Our default setting is marked in gray. A visual illustration of different scan orders are available in the appendix.

🔼 Table 4 presents a comparison of various image generation models on the ImageNet-1K dataset, focusing on 256x256 image generation. The models are categorized by type (diffusion, masked transformer, autoregressive), tokenizer type (discrete VQ or continuous VAE), and whether rejection sampling was used. Results are evaluated using the Fréchet Inception Distance (FID) metric, with additional metrics provided in some cases. Note that some models generate images at a resolution of 384x384 and then resize to 256x256 for consistent evaluation.

read the caption

Table 4: ImageNet-1K 256×256256256256\times 256256 × 256 generation results evaluated with ADM [18]. “type” refers to the type of the generative model, where “Diff.” and “Mask.” stand for diffusion models and masked transformer models, respectively. “VQ” denotes discrete tokenizers and “VAE” stands for continuous tokenizers. “-re” stands for rejection sampling. “-384” denotes for generating images at resolution 384384384384 and resize back to 256256256256 for evaluation, as is used in [52].

🔼 This table compares the speed of generating images (samples/second) using different image generation models on a single NVIDIA A100 GPU. The models are grouped based on their Fréchet Inception Distance (FID) scores, a metric indicating image quality, to ensure a fair comparison. The throughput is measured using float32 precision and a batch size of 128, following the original codebases of each method. Notably, the models using autoregressive architectures (RAR and VAR) utilize KV-cache optimization for efficiency, resulting in higher speeds. ‘Diff.’ indicates diffusion models and ‘Mask.’ represents masked transformer models. The table highlights how the proposed RAR method is not only efficient in generating images but also significantly faster than many other methods with comparable FID scores.

read the caption

Table 5: Sampling throughput comparison (including de-tokenization process) categorized by methods with similar FID scores. Throughputs are measured as samples generated per second on a single A100 using float32 precision and a batch size of 128128128128, based on their official codebases. For VAR [58] and our RAR, KV-cache is applied. “Diff.” and “Mask.” refer to diffusion models and masked transformer models, respectively.

🔼 Table 6 presents the detailed hyperparameter settings used for training the final versions of the Randomized Autoregressive (RAR) models. These settings encompass both training hyperparameters (optimizer, learning rate, weight decay, batch size, learning rate schedule, etc.) and sampling hyperparameters (temperature, scale power, and guidance scale), offering a comprehensive overview of the configuration employed to achieve the reported results. The table is broken down into two sections, one for training and one for sampling, which provides clarity in understanding the various parameters.

read the caption

Table 6: Detailed hyper-parameters for final RAR models.