Exercise 2: What CLIP Actually Does and Why Re-Injection at Every Step Matters — Possible Solution ==================================================================== WHAT CLIP IS TRAINED TO DO ------------------------------ Per this chapter, "CLIP is trained on a separate, simpler task: given a large set of (image, caption) pairs, learn to map both images and text into the SAME numerical vector space, such that a genuinely matching image and caption land close together in that space, and mismatched pairs land far apart." So CLIP's own job has nothing to do with generating images at all — it's purely a matching/embedding task, learning what makes a caption and an image "go together" and encoding that relationship as proximity in a shared vector space. WHAT THE RESULTING TEXT EMBEDDING PROVIDES ------------------------------ Per this chapter, the payoff of that training is "a text embedding — a long list of numbers — that captures, in a form the diffusion network can use, what 'a photo of a golden retriever' means in visual terms." This is the bridge between the two halves of the system described in the chapter: the denoising network (which only understands noisy images and noise predictions) and a text prompt (which is just a string of words) — CLIP's embedding translates the words into the same numerical "visual meaning" space the denoising network can actually condition its predictions on. WHY RE-INJECTING THE EMBEDDING AT EVERY STEP MATTERS ------------------------------ Per this chapter, "that text embedding is fed into the denoising network alongside the noisy image at every single step... The prompt doesn't get read once and remembered — it's re-injected at every denoising step, continually pulling the emerging image toward the text's own embedding." If the embedding were only used once, at the very first step, its influence would have to survive being diluted across every subsequent denoising step with no reinforcement — each step could drift further from what the text actually specified. By re-injecting the same embedding at every single step instead, the text continually corrects and reinforces the direction of generation the whole way through, which is exactly why the final image ends up strongly reflecting the prompt's own descriptive content rather than only loosely resembling it. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains CLIP's own training task precisely as the chapter describes it (a matching task, not a generation task), explains what its output actually is (a shared-space embedding usable by the denoising network), and connects the "re-injected at every step" detail directly to why that produces strong, sustained textual influence over the final image rather than a single early nudge that could fade.