Exercise 3: What loss.backward() and optimizer.step() Actually Do — Possible Solution
====================================================================

loss.backward() → nn1-5's BACKPROPAGATION
------------------------------
Per this chapter's own tip-box, "loss.backward() is nn1-5's own
backpropagation — one line, doing exactly the chain-rule-driven
gradient computation that chapter explained conceptually." Per nn1-5,
backpropagation works by starting at the output layer's own loss,
computing that layer's own gradient, then propagating backward through
every earlier layer using the chain rule, computing each layer's own
contribution to the final loss along the way. When loss.backward() is
called, PyTorch performs exactly this entire process automatically: it
walks backward through the computation graph that was built as the
forward pass executed, applying the chain rule at each step, and stores
the resulting gradient for every single parameter in the network — the
full multi-step algorithm nn1-5 described in prose, executed with one
method call.

optimizer.step() → nn1-6's GRADIENT DESCENT UPDATE
------------------------------
Per this chapter's own tip-box, "optimizer.step() is nn1-6's own
gradient descent update rule, applied automatically." Per nn1-5's own
formula (referenced again in nn1-6's own practical training material),
gradient descent updates each weight according to
new_weight = old_weight − learning_rate × gradient. When
optimizer.step() is called, the optimizer object (here, SGD, configured
with the learning rate 0.1 in this chapter's own code) applies exactly
this update formula to every parameter in the model, using the
gradients that loss.backward() just computed and stored.

WHY THESE TWO CALLS MUST HAPPEN IN THIS SPECIFIC ORDER
------------------------------
optimizer.step() needs the gradients loss.backward() computes in order
to know how much to adjust each weight — calling optimizer.step()
before loss.backward() would have no gradient information available to
act on at all. This is exactly why this chapter's own training loop
calls loss.backward() first, then optimizer.step() immediately after,
mirroring the order nn1-5's own backpropagation description and nn1-6's
own gradient-descent update naturally follow.

WHY optimizer.zero_grad() APPEARS BEFORE BOTH
------------------------------
Though not explicitly asked about, understanding why zero_grad() comes
first reinforces the same point: PyTorch accumulates gradients by
default rather than overwriting them, so without explicitly clearing
old gradients before each new backward() call, gradients from previous
training iterations would incorrectly add onto the current iteration's
own gradients — a real, practical detail underlying why this chapter's
own loop clears gradients at the start of each epoch, before computing
a fresh backward pass.

WHY THIS WORKS AS AN ANSWER
------------------------------
It identifies precisely which earlier chapter each line corresponds to
(nn1-5 for loss.backward(), nn1-6 for optimizer.step()), explains what
each one actually computes using each chapter's own described mechanism,
and explains why the two calls must occur in exactly the order this
chapter's own code shows them.