Stable Diffusion — Open-Source Control & Technical Parameters

imgai1-4's own warn-box drew a real line: Midjourney's parameters can only be described by their observed behavior, because its architecture is never published. Stable Diffusion is the other side of that line. Its model weights, training methodology, and source code are all public — which means this chapter can do something imgai1-4 explicitly couldn't: explain exactly what its own parameters do to the mechanism imgai1-2 already introduced, not just what they're observed to produce.

Local, Self-Hosted, and Genuinely Open

Where Midjourney is cloud-only with no local install (imgai1-4), Stable Diffusion's model weights can be downloaded and run entirely on your own hardware — no account, no subscription, no internet connection required once set up. In practice, most people run it through a community-built interface (Automatic1111's web UI, ComfyUI, or InvokeAI are the most common) rather than writing raw code against the model directly, but the defining fact remains: nothing about running it requires anyone else's server.

CFG Scale — Now Explainable in Full

imgai1-2 previewed classifier-free guidance in one sentence: the network is trained to predict noise both with and without the text embedding, and generation blends the two predictions. Here's the full picture. At every denoising step, the model computes two separate noise predictions from the same noisy image: an unconditional prediction (as if no text prompt existed at all) and a conditional prediction (steered by the text embedding, per imgai1-2's own CLIP mechanism). The final prediction actually used is the unconditional one, plus the difference between the two, amplified by the CFG scale:

final_prediction = unconditional_prediction + CFG_scale × (conditional_prediction − unconditional_prediction)

A CFG scale of 1 uses the conditional prediction roughly as-is. Higher values (commonly 7–12 as a working range) amplify the direction the text embedding is already pulling toward, producing results that adhere to the prompt more strongly. Pushed too high, that amplification overshoots — oversaturated colors, harsh contrast, and visible artifacting, sometimes called "overcooked" output. Lower values produce looser, more creative departures from the literal prompt, at the cost of adherence.

Sampling Steps & Samplers

Steps is simply how many times the reverse denoising loop (imgai1-2) actually runs — more steps generally mean a more fully resolved image, with sharply diminishing returns past a certain point (often somewhere around 20–50, depending on the sampler). Samplers (Euler, DPM++, DDIM, and others) are different numerical algorithms for solving that same denoising process — they don't change what the model has learned, only how efficiently and in what character it converges toward a final image. Different samplers can produce visibly different results from an identical prompt and seed, and some converge acceptably in far fewer steps than others.

Negative Prompts as a First-Class Mechanism

imgai1-4 covered Midjourney's own --no parameter as a simple exclusion list. Stable Diffusion's negative prompt is a genuinely deeper mechanism, and the CFG formula above explains exactly why: instead of using a truly "unconditional" prediction (as if no text existed) as the baseline, a negative prompt is itself run through CLIP the same way the positive prompt is, and its own embedding replaces the unconditional prediction in the formula:

final_prediction = negative_prediction + CFG_scale × (positive_prediction − negative_prediction)

This doesn't just avoid the negative prompt's content — it actively steers generation away from that region of embedding space, at every single denoising step, amplified by the same CFG scale governing how strongly the positive prompt pulls toward its own target. A negative prompt is treated by the mechanism as a full, first-class second prompt, not a simple exclusion filter layered on afterward.

Checkpoints & LoRAs — A Genuinely Unique Capability

Because Stable Diffusion's weights are public, the community has trained thousands of alternate checkpoints — full, re-trained or fine-tuned versions of the base model, often specialized toward a particular art style, subject matter, or level of photorealism. Swapping checkpoints changes the model's own underlying visual "instincts" before a single prompt word is even considered. LoRAs (Low-Rank Adaptation) are smaller, lighter-weight adapter files that can be layered on top of a checkpoint to nudge it toward a narrower style, character, or concept, without the cost of training or storing an entirely new full model. Neither of these has a real equivalent on Midjourney or DALL-E (imgai1-6) — both are closed, single-model, hosted-only systems with nothing analogous to swap in.

Hands-On Exercises