VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generation

Overview

Autoregressive (AR) visual generation relies on tokenizers to map images to and from discrete sequences. However, tokenizers are trained to reconstruct clean images from ground-truth tokens, while AR generators are optimized only for token likelihood. This misalignment leads to generated token sequences that may decode into low-quality images, without direct supervision from the pixel space. Specifically, we propose VA-π, a lightweight post-training framework that directly optimizes AR models with a principled pixel-space objective. VA-π formulates the generator–tokenizer alignment as a variational optimization, deriving an evidence lower bound (ELBO) that unifies pixel reconstruction and autoregressive modeling. To optimize under the discrete token space, VA-π introduces a RL-based alignment strategy that treats the AR generator as a policy, uses pixel-space reconstruction quality as its intrinsic reward. The reward is measured by how well the predicted token sequences can reconstruct the original image under teacher forcing, giving the model direct pixel-level guidance without expensive free-running sampling. The regularization term of the ELBO serves as a natural regularizer, maintaining distributional consistency of tokens.

Observations

1. Alignment to pixel-space

VA-π aligns autoregressive image generation with the ground-truth distribution in pixel space. Qualitatively, VA-π corrects off-manifold token sequences that decode into distorted structures, producing more coherent and faithful reconstructions. Quantitatively, both embedding density estimation (KDE) and low-dimensional projections (t-SNE) show that VA-π shifts generated images closer to the ground-truth manifold.

2. VA-π is better than naive post-train

We benchmark VA-π against naive post-training baselines on two representative settings: class-to-image generation on ImageNet-1K (LlamaGen-XL/XXL; evaluated with and without CFG), and text-to-image reasoning on GenEval (LlamaGen-XL and the unified multimodal model Janus-Pro 1B). We report standard generation metrics (FID/IS/Precision/Recall for C2I; GenEval sub-scores and overall for T2I) and include tuning time and whether an external reward is used.

• Lightweight gains. Without external reward, VA-π improves LlamaGen-XXL on ImageNet-1K from FID 14.36 → 7.65 (w/o CFG) in 25 minutes, and improves LlamaGen-XL from FID 15.55 → 9.23 in 20 minutes.
• Better quality at low cost. VA-π boosts perceptual quality significantly: for LlamaGen-XXL (w/o CFG) IS 86.55 → 116.70, and for LlamaGen-XL (w/o CFG) IS 79.16 → 111.59; with CFG it reaches IS 299.63 on LlamaGen-XL while still keeping tuning time at 20 minutes.
• Generalizes beyond C2I. On GenEval, VA-π improves LlamaGen-XL overall from 0.306 → 0.339, and improves the unified multimodal model Janus-Pro 1B from 0.725 → 0.744, without task-specific fine-tuning on GenEval.

In addition to quantitative improvements, we provide a qualitative comparison to highlight the visual differences among naive post-training baselines and VA-π. Under identical decoding settings (ImageNet-1K, CFG = 1.0), VA-π produces samples with more coherent structure and fewer token-induced artifacts.

For additional qualitative comparisons (C2I and T2I), see More Qualitative Results.

Ablation Study

We ablate key components in VA-π, including reward composition, prior regularization, and contextual noise.

Reward and Loss Composition

Reconstruction-only rewards (L_MSE/L_p) are unstable because the policy drifts from the pre-trained token distribution. Adding token-level prior regularization (L_prior, cross-entropy) stabilizes training, and the full objective achieves the best overall trade-off.

Prior Regularization Term

We vary the regularization weight (β) for KL vs CE variants. Without regularization, optimization diverges (FID 38.63); moderate regularization (β = 0.1) improves both FID and IS, while too-strong regularization (β = 1.0) hurts diversity. CE consistently outperforms KL, with the best results at β = 0.1.

(a) FID over weight β

(b) IS over weight β

Contextual Noise

We ablate the corruption probability (ξ) for contextual noise in the LlamaGen T2I post-train setting. Moderate noise (ξ = 0.5) performs best on GenEval (Overall 0.339), while ξ = 0 or overly strong noise (ξ > 0.75) degrades performance.

More Qualitative Results

Class-to-image generation (ImageNet-1K)

We provide additional qualitative comparisons on C2I generation across ImageNet-1K classes. All samples use identical decoding settings (CFG = 1.0, temperature = 1.0, top-k = 0, top-p = 1.0).

Text-to-image generation (GenEval)

We present additional T2I qualitative comparisons on GenEval prompts, focusing on harder compositional tasks (attribute binding, counting, position, two-object combination). All samples use identical decoding settings (CFG = 5.0, temperature = 1.0, top-k = 0, top-p = 1.0).

Citation

@misc{vapi2025,
            title={VA-$\pi$: Variational Policy Alignment for Pixel-Aware Autoregressive Generation}, 
            author={Xinyao Liao and Qiyuan He and Kai Xu and Xiaoye Qu and Yicong Li and Wei Wei and Angela Yao},
            year={2025},
            eprint={2512.19680},
            archivePrefix={arXiv},
            primaryClass={cs.CV},
            url={https://arxiv.org/abs/2512.19680}, 
      }