Exercise 1: Why the Step Function Worked for nn1-3 But Can't Be Trained by Backpropagation — Possible Solution ==================================================================== WHAT nn1-5's BACKPROPAGATION ACTUALLY NEEDS, PER THIS CHAPTER ------------------------------ Per this chapter, "nn1-5's own backpropagation trains a network by computing gradients — how much a small change in each weight would change the output — and nudging weights in the direction that reduces error." This requires computing a genuine, informative gradient at essentially every point a weight might take. WHY THE STEP FUNCTION FAILS THIS REQUIREMENT ------------------------------ Per this chapter, "the step function is flat everywhere except at one single point, where it jumps discontinuously. Its gradient is zero almost everywhere, and undefined exactly at the jump — there's no useful signal anywhere for gradient-based training to act on." If a weight is nudged slightly and the step function's own output doesn't change at all (because the weighted sum is still on the same flat side of the jump), the computed gradient at that point is exactly zero — telling backpropagation "changing this weight slightly makes no difference," even when, globally, a larger change to that weight absolutely would help. A training algorithm relying on these gradients has no signal telling it which direction, or how far, to move any given weight. WHY THIS DIDN'T STOP nn1-3's OWN WORKED EXAMPLE FROM WORKING ------------------------------ Per this chapter, "nn1-3's own hand-derived weights worked specifically because a human, not gradient descent, chose them." nn1-3's own solution never involved any training process at all — a person who already understood the OR/NAND decomposition simply picked specific numeric weight values directly and verified, by hand, that they produced correct outputs for all four XOR cases. No gradient was ever computed or needed, because no algorithm was searching for those weights — they were supplied ready-made. WHY THIS DISTINCTION MATTERS FOR REAL, PRACTICAL NETWORKS ------------------------------ A real application can't rely on a human hand-deriving the correct weights for every problem the way nn1-3's own small, fully-understood toy example allowed — real problems are far too large and complex for that. Any practically useful network needs an automatic training process (backpropagation) to discover its own weights from data, and that process depends entirely on being able to compute meaningful gradients throughout training. Since the step function provides no such gradients anywhere except at one undefined point, it's simply incompatible with the only realistic way most networks' weights are ever actually determined. WHY THIS WORKS AS AN ANSWER ------------------------------ It explains precisely what backpropagation requires (informative gradients throughout the space of possible weights), explains why the step function structurally cannot provide this (a gradient of zero almost everywhere), and explains why nn1-3's own example avoided this problem entirely by never using gradient-based training in the first place — a distinction that doesn't generalize to real, practically- sized problems.