Exercise 3: What Residual Connections Do Mechanically, and Why They Matter at Depth — Possible Solution ==================================================================== WHAT THE EQUATION x = x + Sublayer(x) ACTUALLY COMPUTES ------------------------------ Rather than a sublayer (attention or feed-forward) completely replacing its own input with its output, the sublayer's output is added to the original input, and that sum becomes the new value of x. This means the original, unmodified input is always still present in the result — the sublayer's own contribution is layered on top of it, not substituted for it. WHY THIS MATTERS FOR GRADIENT FLOW DURING BACKPROPAGATION ------------------------------ Per this chapter's own finding-box, "because the original input x is always present in the sum, gradients during backpropagation have a direct path all the way back through the network via these shortcuts, rather than only flowing through however many transformative sublayers sit in between." When computing how much a very early layer's own parameters contributed to the final output, the residual connection provides one direct additive path back to that early layer — a path that doesn't require passing through, and being potentially shrunk or distorted by, every single transformative sublayer in between. WHY THIS BECOMES ESPECIALLY IMPORTANT AT DOZENS OF STACKED LAYERS ------------------------------ Without residual connections, a gradient computed at the output would have to flow backward through every one of the (per this chapter) up to 96 stacked Transformer blocks in sequence, with each sublayer's own transformation potentially shrinking or distorting that gradient signal along the way. Per nn1-6's own training-stability material, this kind of accumulated degradation across many layers is exactly the mechanism responsible for vanishing gradients in deep networks generally — the deeper the stack, the more opportunities there are for the gradient signal to shrink toward zero before it ever reaches the earliest layers. WHY THE RESIDUAL SHORTCUT SPECIFICALLY COUNTERACTS THIS ------------------------------ Because the residual connection provides a direct additive path alongside each sublayer's own transformative path, a gradient always has at least one route back to any given layer that doesn't get multiplicatively shrunk by that layer's own sublayer computations. This doesn't eliminate the transformative path's own contribution to training — it simply guarantees that even if that path's own gradient signal degrades, the shortcut path keeps a usable gradient flowing, which is precisely why very deep stacks of Transformer blocks remain trainable at all. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely what the addition in `x = x + Sublayer(x)` preserves, connects this directly to how gradients flow during backpropagation, and ties the growing importance of this mechanism at real model depths (dozens of stacked blocks) back to nn1-6's own general vanishing-gradient material, showing why residual connections are a targeted fix for exactly that problem at Transformer scale.