Key takeaways
- KAIST's LOWN skips the image on most decoding steps of diffusion vision-language models and needs no retraining.
- On LLaDA-V it keeps 99.6% of the average benchmark score at 35.3% of the compute, per the authors.
- Savings shrink on chart and document tasks, and the reused image guesses can go stale.
Researchers at KAIST have published LOWN ("Look Only When Needed"), a decoding method that lets a diffusion vision-language model skip looking at the image on most steps. In the paper, posted this week and trending on Hugging Face daily papers, the team says LOWN keeps 99.6% of a model's benchmark score while using 35.3% of the compute, with no retraining.
What a diffusion vision-language model does
Most chat models write text one token at a time, left to right. A diffusion language model works differently. It starts with a row of blank (masked) positions and fills several of them in at once, then repeats until the answer is complete. Each round is called a denoising step.
A diffusion vision-language model, or DVLM, adds an image. The catch is that the model re-reads the whole input on every step, image included. The authors measured the split: image tokens make up 89 to 97% of the input on the benchmarks they studied. So most of the work goes into re-reading a picture that has not changed.
The finding behind the method
The team asked how often the image matters to a given step. They ran LLaDA-V, an open DVLM, with and without the image. Removing the image changed the token being filled in at only 22.8% of steps on average, and at as few as 7.9% (on MMBench). On DocVQA, a document-reading test, the figure was higher, 43.8%. On the whole, the text already written and the earlier guesses carry most of the load.
How LOWN decides when to look
LOWN alternates between two kinds of step:
- A full step reads the image and the text. It saves the model's best guess and confidence for each blank position, called the anchor.
- A text-only step is far cheaper. It guesses the same positions without the image. If those guesses match the anchor, in order of confidence, the longest matching run is "certified."
If the certified run is at least as long as the number of tokens the model would normally fill in that step, LOWN fills in the whole run and skips the image. If not, it pays for a full step and refreshes the anchor. The method changes no weights and needs no tuned threshold.
Results
All numbers below are from the authors' own experiments.
- LLaDA-V, ten image benchmarks: 99.6% of baseline average score at 0.353 times the compute. Among faster methods with at least that much compute, LOWN scored best on 8 of 10 benchmarks. Average generation time fell 57.4%, and by up to 73.2%.
- Against a random schedule with the same number of full steps, LOWN's task-normalized quality was 4.9 to 13.0% higher. A fixed every-fourth-step schedule matching its compute on MathVista scored 4.0 points lower.
- Other models: on LaViDa, 99.7% of baseline at 0.174 times the compute. On the video model VidLaDA-8B, EgoSchema went from 60.6 to 60.2 at 0.255 times the compute, and MLVU went from 41.8 to 43.3 at 0.272 times.
- Stacked with other speedups: combined with visual token pruning (dropping image tokens), it cut the partner method's compute by 57.6% and 60.6% in two tests. Combined with Fast-dLLM, a caching method, it cut generation time by a further 38.1 to 50.1%.
An oracle that used the image only where it truly changed a decision would cost 0.33 times the baseline. LOWN's measured 0.35 is close to that floor.
Limits
The authors list three. A reused anchor can disagree with what a fresh full step would predict, and more often as it ages: 4.1 to 11.0% of the time when it is one step old, up to 25.3% on DocVQA at eight steps. Savings are smaller on charts and documents, where the image keeps mattering. And the total cost cannot be fixed in advance, because the agreement rule decides how many full steps happen.
Combined with Fast-dLLM, scores stayed within 2.0 points of LLaDA-V but fell below Fast-dLLM alone on all three tasks tested, by as much as 5.0 points on LLaVA-Bench. The results come from open research models and from the authors' own tests.
Why it matters
Diffusion language models are still a minority approach, and their appeal rests on speed. Re-reading the image at every step ate into that. If the numbers hold up, a large share of the cost of a DVLM can be removed at inference time, which is the cheap kind of improvement because it works on models that already exist. The paper's page lists code and a project page at hyeon9698.github.io/LOWN. We covered a related diffusion model result in DTCA.
Frequently asked questions
What is a diffusion vision-language model?
It is a model that reads an image and text and writes its answer by filling in many blank positions at once over several rounds, instead of one token at a time.
Does LOWN require retraining the model?
No. The paper describes it as training-free and threshold-free. It changes how the model is run, not its weights.
How much faster is LOWN?
On LLaDA-V the authors report average generation time down 57.4% and up to 73.2%, at 0.353 times the baseline compute.