VAR

Overview of VAR

What it is

VAR is the official implementation of the NeurIPS 2024 Best Paper “Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.” It introduces a new autoregressive paradigm for image generation that performs coarse-to-fine “next-scale prediction” instead of the standard raster-scan “next-token prediction.” This repository provides an ultra-simple, user-friendly codebase for autoregressive image generation and includes pretrained models of varying sizes (from 310M to 2.3B parameters) capable of producing images at 256×256 or 512×512 resolution. The project also reports power-law scaling laws in VAR transformers and for the first time shows GPT-style autoregressive models surpassing diffusion models in visual generation.

Key features

• VAR redefines autoregressive learning on images as next-scale prediction (coarse-to-fine), diverging from traditional next-token prediction.
• For the first time, GPT-style autoregressive models surpass diffusion models in image generation benchmarks.
• Power-law scaling laws are observed in VAR transformers, indicating predictable performance improvements with model size.
• Pretrained models of sizes d16 (310M), d20 (600M), d24 (1.0B), d30 (2.0B), d30-re (2.0B), and d36 (2.3B) are available, with reported FID scores ranging from 1.80 to 3.55 on ImageNet 256×256 or 512×512.
• Zero-shot generalizability is demonstrated across various tasks. The codebase supports automatic resumption of interrupted training and optional flash-attention/xformers for faster attention computation.

Use cases

• Training and evaluating autoregressive image generation models on ImageNet (256×256 or 512×512).
• Reproducing the NeurIPS 2024 Best Paper results and exploring scaling law behaviors.
• Performing zero-shot inference and FID evaluation using pretrained VAR models.
• Extending the VAR framework to text-to-image or text-to-video generation (related projects: Infinity, InfinityStar).

Getting started

See the repository README for full installation and training instructions. Basic steps: 1. Install torch >= 2.0.0. 2. Install other pip packages via pip3 install -r requirements.txt. 3. Prepare the ImageNet dataset at /path/to/imagenet with the expected folder structure. 4. (Optional) Install and compile flash-attn and xformers for faster attention computation. Example training command for VAR-d16 on 256×256: shell torchrun --nproc_per_node=8 --nnodes=... --node_rank=... --master_addr=... --master_port=... train.py --depth=16 --bs=768 --ep=200 --fp16=1 --alng=1e-3 --wpe=0.1

License and pricing

The repository is licensed under MIT. It is open source with no stated pricing.

Based on the project's public README and repository metadata.