How Diffusion Models Actually Work

The History of AI trilogy's own closing chapter, historyai3-8, named "diffusion models" in passing as one of the developments bringing that course up to the present day, alongside multimodal AI and the AGI/alignment debate — a namecheck, not an explanation. This chapter delivers the explanation, and does it specifically because imgai1-1's central claim — that an image prompt is descriptor composition, not instruction-giving — is only a slogan until you can see the actual mechanism that makes it true.

Step One: Learning to Destroy Images

Diffusion models are trained backwards from how you'd expect. Training starts with real images from a large captioned dataset, and a fixed, non-learned process called forward diffusion: a small amount of random (Gaussian) noise is added to the image, repeatedly, over many steps — a few hundred to a few thousand, depending on the model — until the image is statistically indistinguishable from pure noise. This forward process needs no learning at all; it's just repeated noise addition, and it's identical for every image in the dataset.

The interesting part — the part that's actually trained — is the reverse direction: a neural network is trained to look at a noisy image at some step and predict what noise was added, so that subtracting its prediction moves the image one step closer to clean. Do this once, and you've removed a little noise. Do it hundreds of times in sequence, starting from pure random noise instead of a slightly-noised real image, and the network — which has only ever seen "slightly less noisy version of this" as its training signal — ends up hallucinating a coherent image out of static, one small denoising step at a time.

Step Two: Bringing Text Into the Process

Everything above describes an unconditional diffusion model — one that generates images with no text input at all, denoising toward whatever the training data made statistically likely in general. To make text prompting work, the denoising network needs the text to actually steer each denoising step, not just describe the final result after the fact.

This is where CLIP (Contrastive Language-Image Pretraining, an OpenAI model) or a similar text-image embedding model comes in. CLIP is trained on a separate, simpler task: given a large set of (image, caption) pairs, learn to map both images and text into the same numerical vector space, such that a genuinely matching image and caption land close together in that space, and mismatched pairs land far apart. The result is a text embedding — a long list of numbers — that captures, in a form the diffusion network can use, what "a photo of a golden retriever" means in visual terms.

During both training and generation, that text embedding is fed into the denoising network alongside the noisy image at every single step, biasing each prediction toward the region of "clean image space" the embedding points at. The prompt doesn't get read once and remembered — it's re-injected at every denoising step, continually pulling the emerging image toward the text's own embedding.

A Preview of Two Terms This Course Will Need Later

Two mechanical details, previewed here and covered in full technical depth in imgai1-5:

• Latent diffusion: running the denoising process directly on full-resolution pixels is expensive. Stable Diffusion instead compresses images into a smaller "latent" representation first (via a separate encoder network), runs the entire noise/denoise process there, and decompresses back to pixels only at the end — the same mechanism described above, just operating on a compressed representation rather than raw pixels.
• Classifier-free guidance: the network is actually trained to predict noise both with and without the text embedding, and generation blends the two predictions, amplifying the difference the text makes. Stable Diffusion's own CFG scale parameter (imgai1-5) controls exactly how strongly that difference gets amplified.

Why This Mechanism Predicts Real, Later Failure Modes

Hands-On Exercises