SpecSlot cuts pixel-space image compute 41% but trails on quality

SpecSlot is a way to build image-generating Transformers that work directly on pixels without paying full price for fine detail. A paper posted to arXiv on October 29 by researchers at Peking University, SenseTime and Nanyang Technological University describes it in Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers. On ImageNet at 256 pixels, a 138M-parameter SpecSlot model reaches an FID of 6.59 at about 41% less compute than the best baseline it compares against. That baseline is still more accurate, though, and the trade-off is the interesting part.

The problem: more detail means many more tokens

A Transformer sees an image as a list of tokens, usually one per square patch of pixels. Smaller patches keep more detail but make the list longer. For a 256 by 256 image, going from 16-pixel patches to 8-pixel patches raises the count from 256 tokens to 1,024. Attention, the step where every token looks at every other token, grows with the square of the list length, so that change makes the attention term 16 times larger, according to the paper.

The authors argue this is wasteful. Coarse structure, like where the dog is, needs information shared across the whole image. Fine texture is mostly local. Yet a plain patch Transformer gives every token the same global role.

What the team did

SpecSlot splits the job into three steps, which give the paper its name.

• Read. A grid of 256 anchors each gathers evidence from its own cell of the noisy input image.
• Allocate. Each anchor expands into K slots (2, 4 or 8 in the experiments). Each slot is responsible for a separate band of frequencies, using the 2D discrete cosine transform, the same kind of frequency decomposition JPEG uses. One slot takes the lowest frequencies, the others take progressively finer ones.
• Route. Only the lowest-frequency slot from each anchor goes into global attention. The other slots mix only with their neighbors inside the same anchor. The 12-block model alternates two global blocks with two local ones.

So the global sequence on ImageNet-256 holds 288 tokens instead of 1,056, while the output still has the detail of the larger token count. The model still generates in pixel space, trained with rectified flow, a method that learns to move noise toward an image along a straight path. A small spatial head adds a pixel-level correction on top of the frequency-based output.

Results

The paper compares SpecSlot with JiT baselines and with a compute-matched JiT control, all at 12 blocks and hidden width 768. Lower FID is better. Numbers below are the authors' own, averaged over five seeds for the main results.

At 256 pixels, SpecSlot with K=4 beats the compute-matched control (6.59 against 6.68) and the cheap JiT-B/16 (7.51). It does not come close to JiT-B/8, which scores 4.34 but costs about 1.7 times as much. SpecSlot's inception score, a second quality measure where higher is better, is 215.20 against 225.72 for JiT-B/8.

At 512 pixels the gap to the compute-matched control is wider: 9.14 FID for SpecSlot K=4 against 11.98, with about 4% fewer GFLOPs. It still trails the full JiT-B/16 (8.45) by 0.69 FID, at 38.2% lower cost, and the authors report about 1.6 times its throughput.

For text-to-image at 256 pixels, trained on BLIP3o caption shards and JourneyDB, SpecSlot scores 62.21 on DPG-Bench against 61.15 for JiT-B/8, at 40.6% lower cost. On GenEval, JiT-B/8 wins, 29.78 to 26.99.

The ablations point the same way. Routing only the lowest-frequency slot beat routing the others. Letting every token attend globally gave a better FID of 5.84 but cost 225.82 GFLOPs and ran at 5.11 images per second, about half the speed of the chosen design at 9.06.

Limits

This is one preprint with small models, and the comparison is narrow.

• The compute-matched JiT control is the authors' own construction, not a published model, so the headline win is against a baseline they built.
• JiT-B/8 stays ahead on quality at 256 pixels, and JiT-B/16 stays ahead at 512.
• The text-to-image test is a single SpecSlot configuration against two baselines, on two benchmarks.
• Adding slots has diminishing returns. Going from K=4 to K=8 improved FID only from 6.59 to 6.28, and the share of slots the model uses effectively fell from 0.620 to 0.495.
• The authors do not claim the best split between anchors and slots holds beyond the sizes they tried.

Why it matters

Most large image models first compress pictures into a learned latent space and generate there. Working directly in pixels avoids that compression step, but the token count becomes the cost. SpecSlot is an argument that you can decouple the two: keep the global sequence short and spend the extra tokens locally. It does not make pixel-space generation cheap or best in class. It gives a tunable middle point between a short, fast sequence and a long, expensive one, which is how the authors describe it too.

The code, MIT-licensed, is in the SpecSlot repository. Its README says checkpoints are on Hugging Face for two ImageNet models and one text-to-image model, with scripts for training and evaluation on 8 GPUs. Independent replication at larger scale is the thing to watch.