Exercise 2: AlexNet's Three Ingredients, and Where Each Was Introduced — Possible Solution
====================================================================

RELU'S ROLE, PER THIS CHAPTER, INTRODUCED IN nn1-4
------------------------------
Per this chapter, AlexNet used "ReLU (nn1-4) instead of sigmoid/tanh,
avoiding the saturation that would have crippled training at that
depth." Per nn1-4, sigmoid and tanh both saturate at their own extremes
— their gradient approaches zero for large-magnitude inputs, and
per nn1-4's own reasoning, this near-zero gradient severely hampers
backpropagation's ability to update weights. In a deep, 8-layer network
like AlexNet, repeated saturation across many layers would compound
this problem badly. ReLU's own constant gradient of 1 for positive
inputs (nn1-4) avoided this specific failure, making it practical to
actually train a network as deep as AlexNet.

DROPOUT'S ROLE, PER THIS CHAPTER, INTRODUCED IN nn1-6
------------------------------
Per this chapter, AlexNet used "dropout (nn1-6) to control overfitting
in a network with millions of parameters." Per nn1-6, a network with
enormous capacity can memorize its own training data outright, exactly
the failure mode nn1-6 compared directly to ml1-7's own unconstrained
decision tree. AlexNet, with millions of learnable parameters and a
comparatively limited (though large by the standards of its time)
training set, was a genuine, realistic candidate for this kind of
overfitting — dropout's own randomly-thinned-subnetwork mechanism
(nn1-6) helped prevent the network from over-relying on any narrow set
of neurons that happened to fit training-specific quirks.

GPU-BASED MINI-BATCH TRAINING'S ROLE, PER THIS CHAPTER, INTRODUCED IN nn1-6
------------------------------
Per this chapter, AlexNet was trained "on GPUs using mini-batches
(nn1-6), making a network of that depth practically trainable within a
realistic timeframe at all." Per nn1-6, mini-batch gradient descent
specifically enables efficient parallel computation on GPU hardware —
without this, training a network as large and deep as AlexNet using
either pure batch gradient descent (impractically slow, one update per
full pass over a huge dataset) or pure stochastic gradient descent (no
meaningful use of parallel hardware) would have been far less
practically feasible within any reasonable amount of time.

WHY ALL THREE WERE NECESSARY, NOT JUST ONE
------------------------------
Per this chapter's own finding-box framing, these are named together as
"three specific technical choices, each one this course has now already
covered, [that] are real, documented reasons it succeeded where
earlier, shallower attempts hadn't." Each ingredient addresses a
genuinely different obstacle to training a deep network successfully
(a gradient-flow problem, an overfitting problem, and a raw
computational-feasibility problem) — removing any one of the three
would have left a different, still-serious obstacle unaddressed, which
is why the chapter credits the combination rather than any single
technique alone.

WHY THIS WORKS AS AN ANSWER
------------------------------
It identifies precisely what each of the three named techniques
addressed (saturation/gradient flow, overfitting, and computational
feasibility), traces each one back to its specific originating chapter
(nn1-4 for ReLU, nn1-6 for dropout and GPU mini-batch training), and
explains why AlexNet's success required all three together rather than
any one in isolation.