Neighboring Autoregressive Modeling for Efficient Visual Generation

🔼 Figure 2 presents a comparison of the generation quality and efficiency of different visual generation models. The models all have approximately 300 million parameters. The comparison is based on results from the ImageNet 256x256 dataset. The figure shows that NAR-M achieves a balance between high image quality and high generation speed, outperforming other models.

read the caption

Figure 2: Generation quality and efficiency comparisons between various visual generation methods. Data is collected from ImageNet 256×256256256256\times 256256 × 256 dataset over models with parameters around 300M.

🔼 This figure compares different autoregressive visual generation methods, highlighting their generation processes. It illustrates how traditional methods (AR, MAR, PAR, VAR) generate images in raster or random order, often lacking spatial coherence and efficiency. In contrast, the proposed NAR (Neighboring Autoregressive) method progressively generates the image like an ‘outpainting’, starting from an initial token and expanding outwards. This approach is shown to improve efficiency and maintain spatial consistency, as adjacent pixels are generated before more distant pixels.

read the caption

Figure 3: Comparisons of different autoregressive visual generation paradigm. The proposed NAR paradigm formulates the generation process as an outpainting procedure, progressively expanding the boundary of the decoded token region. This approach effectively preserves locality, as all tokens near the starting point are consistently decoded before the current token.

🔼 This figure illustrates the architecture of the dimension-oriented decoding heads in the Neighboring Autoregressive Modeling (NAR) framework. Two heads process the input tokens in parallel. The horizontal head predicts the next token along the row dimension (horizontally), while the vertical head predicts the next token along the column dimension (vertically). Both heads utilize a backbone network consisting of L Transformer blocks, which process the input embeddings to generate the predictions. This parallel processing significantly speeds up inference by predicting multiple tokens simultaneously.

read the caption

Figure 4: Illustration of the dimension-oriented decoding heads. The horizontal head and the vertical head are responsible for predicting the next token in the row and column dimensions, respectively. Here, L𝐿Litalic_L is the number of Transformer blocks in the backbone network.

🔼 This figure illustrates the attention masks used in the Neighboring Autoregressive Modeling (NAR) framework. It highlights how the model handles spatial relationships between image tokens during parallel generation. The different colors represent tokens generated simultaneously in the same step, demonstrating the parallel aspect of NAR. The causal masking ensures the autoregressive property of the model, preventing future tokens from influencing the generation of past tokens. Bidirectional attention within each step fosters consistency by allowing tokens within the same generation step to interact with each other, regardless of their position relative to the starting point.

read the caption

Figure 5: Proximity-aware attention masks for the NAR paradigm. “Sn𝑛nitalic_n” denotes the n𝑛nitalic_n-th generation step. Tokens generated within the same step are represented by the same color. To maintain the autoregressive property, a causal mask is applied between tokens across different generation steps (aligned with Figure 3). Within each step, bidirectional attention is employed among the tokens to enhance consistency during parallel generation.

🔼 Figure 6 presents a comparison of the efficiency of three different autoregressive image generation methods: vanilla AR, VAR, and the proposed NAR. The comparison is shown across three metrics: latency (the time it takes to generate an image), GPU memory usage, and throughput (images generated per second). For batch sizes exceeding 64, NAR demonstrates superior performance, achieving a lower FID (Fréchet Inception Distance, a measure of image quality), reduced latency, lower memory consumption, and substantially increased throughput. This indicates that NAR offers a more efficient and effective approach to autoregressive image generation, especially when dealing with larger batch sizes.

read the caption

Figure 6: Efficiency comparisons between vanilla AR, VAR and the proposed NAR paradigm for visual generation. With a batch size larger than 64, NAR achieves a lower FID with lower latency, lower memory usage and significantly higher throughput.

🔼 This figure visualizes video generation results from the NAR-XL model on the UCF-101 dataset. Each row presents a sequence of 16 frames from a short video, each frame sized at 128x128 pixels. The videos represent diverse action categories, showcasing the model’s ability to generate coherent and visually consistent videos across different actions.

read the caption

Figure A: Video generation samples on UCF-101 dataset. Each row shows sampled frames from a 16-frame, 128×128128128128\times 128128 × 128 resolution sequence generated by NAR-XL across various action categories.

🔼 This figure shows several example images generated by the NAR-XXL model, demonstrating its ability to generate images conditioned on a class label. Each set of images shows different variations of the same object class generated from the same class label. The classes shown include siamese cat, coral reef, volcano, lesser panda, valley, great white shark, daisy, and geyser. This showcases the model’s capacity to produce visually diverse outputs belonging to the same class.

read the caption

Figure B: Class-conditional image generation samples produced by NAR-XXL on ImageNet 256×256256256256\times 256256 × 256.

🔼 This figure showcases several examples of images generated by the NAR-XXL model (Neighboring Autoregressive Model, extra-large size). Each example demonstrates the model’s ability to generate high-quality, class-conditional images of 256x256 pixels. The images are grouped by class (e.g., cheeseburger, ice cream, schooner), showing the variety and detail within each class that the model produces.

read the caption

Figure C: Class-conditional image generation samples produced by NAR-XXL on ImageNet 256×256256256256\times 256256 × 256.

🔼 This figure displays a comparison of image generation results between two models: LlamaGen-XL-Stage1 and NAR-XL-Stage1. Both models generated 256x256 pixel images based on text prompts, but they employed different generation methods. LlamaGen-XL-Stage1 utilizes the standard ’next-token’ prediction approach, sequentially predicting each pixel one by one. In contrast, NAR-XL-Stage1 employs a more efficient ’next-neighbor’ prediction approach, which allows parallel processing and reduces the number of steps needed for generation. The figure visually showcases the differences in image quality and generation efficiency between these two methods.

read the caption

Figure D: 256×256256256256\times 256256 × 256 text-guided image generation samples produced by LlamaGen-XL-Stage1 with next-token prediction paradigm and NAR-XL-Stage1 with next-neighbor prediction paradigm.

🔼 Figure E shows a comparison of image generation results between LlamaGen-XL-Stage2 (a model using the traditional next-token prediction approach) and NAR-XL-Stage2 (a model utilizing the novel next-neighbor prediction paradigm proposed in the paper). Both models were tasked with generating 512x512 pixel images based on text prompts sourced from the Parti dataset. The figure visually demonstrates the differences in image quality and the efficiency of the two methods.

read the caption

Figure E: 512×512512512512\times 512512 × 512 text-guided image generation samples produced by LlamaGen-XL-Stage2 with next-token prediction paradigm and NAR-XL-Stage2 with next-neighbor prediction paradigm. The text prompts are sampled from Parti prompts.

🔼 Table 2 presents a comparison of different class-conditional video generation methods evaluated on the UCF-101 benchmark. The table focuses on the efficiency and quality of these methods. Key metrics compared include the Frechet Video Distance (FVD), which measures the difference between generated and real videos, the number of steps required for video generation, and the total time taken for the process. The table also includes model parameters for each method to contextualize the performance differences. A note indicates that a particular model (LP) is a variant of the XL model, but it’s smaller due to the removal of six layers, allowing for a comparison of performance with a reduced model size.

read the caption

Table 2: Comparison of class-conditional video generation methods on UCF-101 benchmark. Classifier-free guidance is set to 1.25 for all variants of our method. ††\dagger†: model denoted as LP shares the same hidden dimension as the XL model but is reduced by 6 layers in depth.

🔼 Table 3 presents a quantitative comparison of different models on the GenEval benchmark, a standardized evaluation for text-to-image generation. The table shows how various models perform across different aspects of image generation, such as the ability to generate images corresponding to counts, objects (single and two), and colors specified in the text prompts. It also reports overall performance scores, providing a comprehensive assessment of each model’s accuracy and effectiveness in understanding and translating textual descriptions into visual representations. The results highlighted in this table are particularly useful for comparing the performance of different models on a diverse range of text-to-image generation tasks.

read the caption

Table 3: Quantitative evaluation on the GenEval benchmark. ††\dagger† denotes the results are reported by [16].

🔼 This table presents the ablation study results on the effect of using dimension-oriented decoding heads in the NAR model for image generation. It compares the performance of the NAR model with and without dimension-oriented decoding heads, evaluated on the ImageNet 256x256 benchmark. The results show the FID scores and the number of steps taken for generation. The ‘NTP’ (next-token prediction) paradigm is used as a baseline for comparison. The comparison highlights the impact of the proposed dimension-oriented heads on the model’s performance in terms of both image quality (FID) and efficiency (number of steps).

read the caption

Table 4: The effect of dimension-oriented decoding heads evaluated on ImageNet 256×256256256256\times 256256 × 256 benchmark. Here, “NTP” denotes the next-token prediction paradigm.

🔼 This table presents a comparison of different configurations for decoding heads in the Neighboring Autoregressive Modeling (NAR) method, specifically focusing on their impact on the quality of image generation on the ImageNet 256x256 dataset. The experiment uses a classifier-free guidance of 2.0. The configurations vary in which decoding heads are used for overlapping tokens (i.e., tokens that are predicted by more than one head). The table shows the FID (Fréchet Inception Distance) and IS (Inception Score) for each configuration, lower FID indicating better quality.

read the caption

Table 5: Performance comparison of various decoding head configurations on ImageNet 256×256256256256\times 256256 × 256. Here, “Head” denotes the decoding heads to use for the overlapped region. Results are obtained with a classifier-free guidance of 2.0.