# What is a diffusion model: AI that turns noise into images and text

> Diffusion models learn to undo noise step by step. The idea powers Stable Diffusion and is now being tried for language in models like LLaDA and Gemini Diffusion.

URL: https://claudegpt.si/news/what-is-a-diffusion-model  
Published: 2026-11-18T11:06:39.745Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: Diffusion models, Stable Diffusion, Hugging Face, Google DeepMind, Image generation, Research  
Publisher: ClaudeGPT Si

## Key takeaways

- A diffusion model is trained to remove noise step by step, and generates by starting from pure noise and denoising until an image or text appears.
- Latent diffusion, the design behind Stable Diffusion, runs the denoising in a compressed latent space instead of on pixels to cut compute.
- Diffusion language models such as LLaDA 8B fill in masked tokens in parallel instead of writing one token at a time.

A diffusion model is a generative AI model that learns to turn random noise into data, one small cleanup step at a time. Here is what that means in practice: during training it watches real images (or sentences) get buried in noise and learns to undo the damage. To make something new, it starts from pure noise and runs that cleanup over and over until a picture, a video frame or a block of text comes out.

That one idea sits behind Stable Diffusion and the image and video pipelines in Hugging Face's Diffusers library, and it is now being tried on language too. Below is how it works, where it came from, how it differs from the large language models behind chatbots, and what people mean when they compare a "diffusion model" with a "checkpoint".

## How a diffusion model works

There are two processes, one fixed and one learned.

The forward process destroys data. Take a training image and add a little Gaussian noise (random static). Do it again, and again. After enough steps the image is indistinguishable from static. Nothing is learned here; it's a fixed recipe.

The reverse process rebuilds data. A neural network is shown a noisy image and the step number, and is trained to predict what the slightly cleaner image looked like. Once it can do that at every noise level, you can chain the steps backward: start with static, denoise, denoise, denoise, and an image appears that was never in the training set.

The numbers from the paper that made this work well give a sense of scale. In [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) (Ho, Jain and Abbeel, 2020), the forward process ran for 1,000 steps, with the amount of noise per step rising linearly from 0.0001 to 0.02. The network, a U-Net (an image network that shrinks the picture down and builds it back up), was trained on a simple goal: look at a noisy image and predict the exact noise that was added. Subtract that, and you are one step closer to a clean image.

The catch is speed. Generating means running the network once per step. The DDPM authors report that on a TPU v3-8, sampling a batch of 256 small 32-by-32 CIFAR-10 images took 17 seconds, and a batch of 128 images at 256 by 256 took 300 seconds. Much of the work since then has gone into getting good results in fewer steps.

## Where the idea came from

The idea is older than the image boom. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan and Surya Ganguli described it in March 2015 in [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://arxiv.org/abs/1503.03585). Borrowing from physics, they proposed slowly destroying the structure in data with a diffusion process, then learning a reverse process that restores it. Their argument was that this gives a model flexible enough to fit complex data and still simple enough to train and sample from, even with thousands of time steps.

It took five years for the results to catch up. The 2020 DDPM paper reported an FID score of 3.17 on unconditional CIFAR-10, the best reported at the time (FID measures how close generated images are to real ones; lower is better), and an Inception score of 9.46. On 256-by-256 LSUN images, the authors say quality was similar to ProgressiveGAN, then one of the best image generators. The code is [on GitHub](https://github.com/hojonathanho/diffusion).

## Latent diffusion: why image models got practical

Running 1,000 denoising steps on full-resolution pixels is expensive. The authors of [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) (Rombach, Blattmann, Lorenz, Esser and Ommer, CVPR 2022) note that pixel-space diffusion models often take hundreds of GPU days to train.

Their fix was to stop working on pixels. First, an autoencoder learns to squeeze an image into a much smaller grid of numbers, called a latent, and to expand it back. Then the diffusion model does all its noising and denoising in that small latent space. At the end, the autoencoder's decoder turns the clean latent into a full image. The paper splits the labor this way: the autoencoder throws away fine detail the eye barely notices, and the diffusion model learns what the picture is about.

They tested shrink factors from 1 (plain pixels) to 32 and found that downsampling by 4 or 8 gave the best results. They also added cross-attention layers so the model could follow a condition such as a text prompt. Their text-to-image model had 1.45 billion parameters and was trained on LAION-400M.

This is the design behind Stable Diffusion. The [CompVis Stable Diffusion repository](https://github.com/CompVis/stable-diffusion) calls it "a latent text-to-image diffusion model" that builds on the latent diffusion paper. It pairs a factor-8 autoencoder with an 860-million-parameter U-Net and a 123-million-parameter CLIP text encoder, and the README says it runs on a GPU with at least 10 GB of memory.

## What a diffusion model looks like in code

Hugging Face's [Diffusers library](https://huggingface.co/docs/diffusers/index) is the most common way to run open diffusion models in Python. Its docs make it clear that a "diffusion model" is really a pipeline of several parts:

- **The denoiser**: a U-Net, or in newer models a diffusion transformer (DiT). This is the network that predicts noise at each step.
- **A text encoder**: turns the prompt into numbers the denoiser can attend to.
- **A VAE**: the autoencoder that moves between pixels and the latent space.
- **A scheduler**: the algorithm that decides how much noise to remove at each step. The [scheduler docs](https://huggingface.co/docs/diffusers/api/schedulers/overview) say it adds noise during training and, during inference, updates the sample from the model's output. Swapping schedulers trades speed against quality.

Loading one takes a few lines. This example is from the Diffusers [loading guide](https://huggingface.co/docs/diffusers/using-diffusers/loading):

```python
import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
  "Qwen/Qwen-Image", dtype=torch.bfloat16, device_map="cuda"
)
```

`DiffusionPipeline` reads the repository's `model_index.json` and picks the right pipeline class for the model.

## Diffusion model vs checkpoint

People who use Stable Diffusion tools such as ComfyUI or Automatic1111 often see "model" and "checkpoint" used for the same download. They are related but not the same thing.

The diffusion model is the architecture and method: the denoiser, the VAE, the text encoder, the scheduler and the training recipe. A checkpoint is a saved file of trained weights for that architecture. Many checkpoints can share one architecture. A fine-tune of Stable Diffusion XL, for example, is a different checkpoint of the same kind of model.

The [Diffusers model formats guide](https://huggingface.co/docs/diffusers/using-diffusers/other-formats) describes two ways checkpoints are stored:

| Format | What's in it | Typical use |
| --- | --- | --- |
| Diffusers format | A folder with `model_index.json` and one subfolder each for the U-Net or transformer, text encoder, VAE and scheduler | `from_pretrained()` in Diffusers |
| Single-file format | All the weights in one `.safetensors` or `.ckpt` file | ComfyUI, Automatic1111, `from_single_file()` in Diffusers |

The file type matters too. The guide recommends `.safetensors`, which Diffusers loads by default, and warns that older `.ckpt` files are saved with Python's pickle and "may be unsafe" because they can be exploited to run malicious code. If you download checkpoints from strangers, prefer safetensors.

## Diffusion language models

Text is made of discrete tokens, not pixels, so you can't sprinkle Gaussian noise on a sentence in the same way. The common workaround is masking. In the forward process, words are hidden behind a mask token, more at each step, until the whole answer is blank. The model learns to fill in the blanks.

The clearest example is [LLaDA](https://arxiv.org/abs/2502.09992) (Large Language Diffusion Models), from Shen Nie, Chongxuan Li and colleagues, first posted in February 2025. They trained an 8-billion-parameter masked diffusion model from scratch on 2.3 trillion tokens, using 0.13 million H800 GPU hours. To answer a prompt, LLaDA starts with a fully masked response, predicts every masked token at once, keeps the ones it is most confident about, re-masks the rest and repeats. The number of steps is a setting that trades speed for quality.

The authors report that LLaDA 8B is competitive with LLaMA3 8B at in-context learning and performs comparably to autoregressive baselines they trained themselves. They also list its limits: the answer length has to be set in advance, it had not been tuned with reinforcement learning, and it is smaller and trained on less data than leading chat models. Our article on [DTCA](https://claudegpt.si/news/dtca-diffusion-language-model-rl) covers one later attempt to add reinforcement learning to LLaDA-8B-Instruct for math.

Google DeepMind has an experimental version too. Its [Gemini Diffusion](https://deepmind.google/models/gemini-diffusion/) page describes a text diffusion model that learns "by refining noise, step-by-step" and can generate "entire blocks of tokens at once". Google reports an average sampling speed of 1,479 tokens per second, plus 0.84 seconds of overhead. Its [May 20, 2025 announcement](https://blog.google/innovation-and-ai/models-and-research/google-deepmind/gemini-diffusion/) says the model turns random noise into text or code, similar to how Google's image and video generators work. It is available only as an experimental demo.

## Diffusion model vs LLM

Standard LLMs are autoregressive. As Google's Gemini Diffusion page puts it, traditional language models generate text one token at a time, so each token waits on the one before it. Many are built on transformers, which our explainer on [transformer models](https://claudegpt.si/news/what-is-a-transformer-model) covers.

A diffusion language model drafts the whole answer and refines it in parallel over a fixed number of steps. That's where the speed claims come from, and it lets the model revise earlier tokens instead of being stuck with them. The trade-off, so far, is that the published diffusion LLMs are smaller and less tested than the best autoregressive ones. The two aren't opposites, either: LLaDA uses a transformer to predict its masked tokens. "Diffusion" describes how the model generates, and "transformer" describes the network doing the predicting.

## Diffusion and this week's image launches

Image and video models launch every week, and few closed ones say what is inside. Microsoft introduced [MAI Image 2.7](https://microsoft.ai/news/introducing-mai-image-2-7/) and MAI Image 2.7 Flash on November 17, 2026, in public preview on Microsoft Foundry at about $35 per 1,000 images for the full model and about $10 for Flash. Microsoft says it ranks first on Artificial Analysis's image-editing leaderboard as of November 16. The announcement doesn't say whether the models use diffusion or describe their architecture at all.

Kling Image 4.0 has drawn coverage on third-party sites, but a search on November 18 turned up no official Kling announcement or model card for it, so there is nothing confirmed to say about how it works.

That gap is common. With open models such as Stable Diffusion, the paper, code and weights show the method. With closed products, take the word "diffusion" only from the company itself.

## What diffusion models still struggle with

Sampling is slow by nature. The latent diffusion paper says its sequential sampling is still slower than GANs, and each step is a full pass through the network. Latent models also inherit the autoencoder's limits: the paper says reconstruction quality can bottleneck tasks that need fine pixel-level accuracy. For text, the open questions are scale and tuning. LLaDA's authors call their fast sampling methods preliminary and say alignment with reinforcement learning is still to come.

## Frequently asked questions

### What is a diffusion model in AI?

It is a generative model trained to reverse a process that gradually adds noise to data. To create something, it starts from random noise and removes a little at each step until a new image, video frame or text appears.

### What is the difference between a diffusion model and a checkpoint?

The diffusion model is the architecture and method: denoiser, VAE, text encoder and scheduler. A checkpoint is a saved file of trained weights for that architecture, stored either as a Diffusers folder or as a single .safetensors or .ckpt file.

### Is an LLM a diffusion model?

Standard LLMs are autoregressive and generate one token at a time. Diffusion language models such as LLaDA and Google's experimental Gemini Diffusion instead start from a fully masked or noisy answer and refine all of it over several steps.

### Is Stable Diffusion a diffusion model?

Yes. Its repository describes it as a latent text-to-image diffusion model that builds on the 2022 latent diffusion paper by Rombach and colleagues.

## Sources

- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://arxiv.org/abs/1503.03585), arXiv (Sohl-Dickstein et al.)
- [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239), arXiv (Ho, Jain and Abbeel)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752), arXiv (Rombach et al.)
- [Large Language Diffusion Models](https://arxiv.org/abs/2502.09992), arXiv (Nie et al.)
- [Stable Diffusion repository](https://github.com/CompVis/stable-diffusion), CompVis
- [Diffusers documentation](https://huggingface.co/docs/diffusers/index), Hugging Face
- [DiffusionPipeline loading guide](https://huggingface.co/docs/diffusers/using-diffusers/loading), Hugging Face
- [Diffusers model formats](https://huggingface.co/docs/diffusers/using-diffusers/other-formats), Hugging Face
- [Diffusers schedulers](https://huggingface.co/docs/diffusers/api/schedulers/overview), Hugging Face
- [Gemini Diffusion](https://deepmind.google/models/gemini-diffusion/), Google DeepMind
- [Gemini Diffusion is our new experimental research model](https://blog.google/innovation-and-ai/models-and-research/google-deepmind/gemini-diffusion/), Google
- [Introducing MAI Image 2.7](https://microsoft.ai/news/introducing-mai-image-2-7/), Microsoft AI
