You already built a neural network's most basic building block. Not metaphorically — literally. This chapter proves it, then shows exactly what's new once you stop stopping at one.

One Neuron Is ml1-5's Own Logistic Regression, Unchanged

ml1-5's own logistic regression: take a weighted sum of the inputs, add a bias, squash the result through sigmoid.

ml1-5's logistic regression:
z = w1*x1 + w2*x2 + ... + wn*xn + b
output = sigmoid(z)

A single artificial neuron computes exactly this — no more, no less:

a single neuron:
z = w1*x1 + w2*x2 + ... + wn*xn + b
output = activation(z)

So What's Actually New?

Two genuinely new ideas, both architectural rather than mathematical:

• Many neurons, side by side. Instead of one neuron producing one output, a layer runs several neurons in parallel against the same inputs — each with its own independently learned weights and bias, each producing its own output.
• Layers, stacked. One layer's outputs become the next layer's inputs. A layer sitting between the raw inputs and the final output is a hidden layer — inputs, hidden layers, and outputs together form the network.

Terminology, Before It's Needed

Why Bother Stacking Layers At All?

A single neuron — one weighted sum, one squash — can only ever draw a straight decision boundary through its inputs, a flat line or hyperplane. No amount of retuning its weights changes that basic geometric limit. nn1-2 shows a real, concrete problem a single neuron structurally cannot solve, no matter how it's trained. nn1-3 shows exactly why stacking a hidden layer underneath fixes it. This chapter deliberately doesn't resolve that mystery yet — it's worth sitting with the question first.

This Course's Own Roadmap

Hands-On Exercises

historyai3-3 named "the Perceptron, Minsky/Papert's critique" as part of its own narrated history of AI, without stopping to explain the actual mechanism or the actual mathematical limit involved. This chapter delivers both, technically — and delivers the real, concrete proof of nn1-1's own closing claim: a single neuron can only draw a straight line.

Rosenblatt's Perceptron (1958)

Frank Rosenblatt's perceptron is, in this course's own terms, exactly nn1-1's single neuron — one weighted sum, one threshold decision, trained via a simple rule: when the perceptron misclassifies an example, nudge each weight slightly in the direction that would have made the correct answer more likely. Real, working, and genuinely exciting for its era — contemporary press coverage reported startlingly ambitious claims about what the technology would eventually achieve, including the U.S. Navy's own widely reported 1958 expectation that it would one day "walk, talk, see, write, reproduce itself and be conscious of its existence."

What the Perceptron Could Actually Do

For a linearly separable problem — one where a single straight line can separate the two classes — the perceptron's own learning rule reliably finds such a line, converging to a correct classifier. This genuinely worked for many real problems. The trouble was never training within that limit; it was the limit itself.

The XOR Problem — A Real, Provable Impossibility

XOR ("exclusive or") takes two binary inputs and outputs 1 if exactly one of them is 1, 0 otherwise:

Plot these four points on a simple 2D grid. The two points that should output 1 — (0,1) and (1,0) — sit on one diagonal. The two that should output 0 — (0,0) and (1,1) — sit on the other diagonal. No single straight line can put both diagonal pairs on their own correct side simultaneously — try to draw one, and at least one point always ends up misclassified.

Minsky & Papert, 1969 — The Real Critique

Marvin Minsky and Seymour Papert's book Perceptrons formally proved the XOR limitation (and others like it) for single-layer networks. The honest, often-missed nuance: the book didn't claim a multi-layer network could never overcome this — it specifically noted the open question of whether an effective training method for multi-layer networks even existed. At the time, it didn't, in any widely practical form. nn1-5 covers backpropagation — the training method that eventually filled that exact gap, in 1986.

What Comes Next

nn1-1 already named the fix in outline: stack a hidden layer underneath. nn1-3 shows exactly why that works — a real, worked demonstration that a two-layer network genuinely does solve XOR, geometrically. This chapter deliberately stops short of showing that solution.

Hands-On Exercises

nn1-2 proved, geometrically, that no single straight line separates XOR. This chapter builds an actual, working two-layer network that solves it — and shows the general principle that makes the solution possible, not just this one specific case.

The Core Idea: A Hidden Layer Transforms the Space

A hidden layer doesn't just add more tunable numbers — it re-represents the input as a new set of coordinates, computed by the hidden neurons themselves. A problem that's unsolvable by a straight line in the original input space can become solvable by a straight line in this new, transformed space. That's the entire trick depth relies on.

A Real, Worked XOR Solution

XOR can be built from two simpler, individually linearly-separable pieces: XOR(x1, x2) = OR(x1, x2) AND NAND(x1, x2) — true exactly when at least one input is 1, and not both are 1. Both OR and NAND are themselves linearly separable (a single line can solve each one alone). One concrete set of weights that implements this — one valid solution among several that would work:

Hidden neuron 1 (approximates OR):   step(x1 + x2 - 0.5)
Hidden neuron 2 (approximates NAND): step(-x1 - x2 + 1.5)
Output neuron (AND of both hidden outputs): step(h1 + h2 - 1.5)

Why This Requires Genuine Nonlinearity

This trick only works because the hidden layer's own activation function (step above; nn1-4 covers the real options) is nonlinear. Stack two purely linear layers with no nonlinearity between them, and the math collapses: a linear function of a linear function is still just one linear function — mathematically indistinguishable from a single layer, with none of this chapter's own transformation trick available at all. nn1-4 picks this up directly.

Closing nn1-2's Own Historical Thread

This is, in principle, exactly the resolution Minsky and Papert's own book left open in nn1-2 — a multi-layer network genuinely can solve XOR. What this chapter's own worked example doesn't do is learn these weights automatically from data the way ml1-3's own regression or nn1-2's own perceptron rule did — these specific numbers were hand-derived for this one small, known problem. A real, general, automatic training method for networks like this one didn't arrive until 1986 — nn1-5's own backpropagation, the actual historical breakthrough that made deep networks practical rather than hand-built curiosities.

Beyond XOR — Why This Is "Depth," Not Just "Size"

The same principle scales: each additional layer can compose the previous layer's own transformed representation into something even more re-arranged, letting a network build up genuinely hierarchical features — this is the real, technical meaning behind "deep" in deep learning, and it's a fundamentally different lever from simply adding more neurons to one single layer (more width, same transformation depth).

Hands-On Exercises

nn1-3 used the step function for a reason — it's the simplest possible illustration of nonlinearity. It's also not what real networks actually use. This chapter covers the real options, and a real problem that follows directly from the choice.

Why the Step Function Doesn't Actually Work in Practice

nn1-5's own backpropagation trains a network by computing gradients — how much a small change in each weight would change the output — and nudging weights in the direction that reduces error. The step function is flat everywhere except at one single point, where it jumps discontinuously. Its gradient is zero almost everywhere, and undefined exactly at the jump — there's no useful signal anywhere for gradient-based training to act on. nn1-3's own hand-derived weights worked specifically because a human, not gradient descent, chose them.

Sigmoid's Own Real Problem — Saturation

For very large positive or very large negative inputs, sigmoid's own curve goes nearly flat — its gradient (slope) approaches zero in those regions, a state called saturation. A neuron whose weighted sum regularly lands far out in either flat tail produces almost no useful gradient for nn1-5's own backpropagation to work with — the weight barely updates at all, regardless of how wrong the prediction actually is.

Tanh — The Same Problem, Recentered

Tanh's own zero-centered output is a real, practical improvement: sigmoid's outputs are always positive, which can bias how gradients accumulate across a whole layer in later training steps; tanh's own (-1, 1) range avoids that specific bias. But tanh still saturates at its own extremes — the same flattening, the same near-zero gradient far from center. Recentering the output doesn't remove the underlying saturation problem, just one specific side effect of it.

ReLU — Simpler, and a Real Fix for Part of the Problem

max(0, x): zero for any negative input, exactly itself for any positive input. For every positive input, the gradient is a constant 1 — no saturation at all on that side. This genuinely helped make much deeper networks practically trainable, a real, documented factor alongside other advances behind the deep-learning resurgence historyai3-3 already named — the 2006 deep-belief-network revival and the years of architecture improvements that followed it.

Choosing an Activation Function in Practice

Hands-On Exercises

nn1-3 proved a multi-layer network can solve XOR, using weights a human worked out by hand. historyai3-3 named "1986 backpropagation" as part of AI's own narrated history without ever explaining the mechanism. This chapter delivers both — the real algorithm that finds those weights automatically, for any network, from data alone.

Forward Propagation — Formally Naming What Every Example Already Did

Forward propagation is simply running the network forward: input values flow into the first layer, each neuron computes its weighted sum plus activation, those outputs become the next layer's inputs, and so on until the final output layer produces a prediction. Every worked example since nn1-1 has already done this — this chapter just gives it a name before building on it.

Loss Functions — Cross-Entropy, Classification's Own Counterpart to MSE

ml1-4's own MSE measured regression error. Classification networks typically use a different loss: cross-entropy, which specifically measures how far a predicted probability distribution sits from the true, correct answer.

The Chain Rule — How Blame Travels Backward

A multi-layer network is a chain of functions — one layer's output feeding the next. If a small change to an early-layer weight changes that layer's output, which changes the next layer's output, which changes the final loss, the total effect of that one weight on the final loss is the product of each individual step's own local effect, chained together — precisely the calculus chain rule. Conceptually: if A affects B, and B affects C, then A's own effect on C is A's effect on B, multiplied by B's effect on C.

Backpropagation — The Real Algorithm

This is the real, 1986 breakthrough (Rumelhart, Hinton, and Williams) — a general, automatic method for training a network of any depth, not a hand-derivation limited to one small, already-understood problem like nn1-3's own XOR example. This is the piece Minsky and Papert's own 1969 book (nn1-2) left as an open question, resolved for real seventeen years later.

Gradient Descent — The Actual Update Rule

new_weight = old_weight - learning_rate * gradient

The learning rate controls how large each step is. Too high, and weight updates overshoot, bouncing past good solutions or diverging entirely; too low, and training crawls, taking an impractically long time to converge. nn1-6 covers this, and several real variants of the basic gradient-descent rule, in practical depth.

Closing nn1-3's Own Deferred Promise

Hands-On Exercises

nn1-5 gave the mechanism. This chapter covers how it's actually run at real scale, and how to stop a network with enormous capacity from doing exactly what ml1-7's own unconstrained decision tree did — memorizing the training set instead of learning from it.

Batch, Stochastic & Mini-Batch Gradient Descent

Mini-batch is the near-universal real-world default specifically because a batch of, say, 64 examples can be processed simultaneously on GPU hardware — the same parallel-computation advantage nn1-10's own framework tour will cover directly.

Epochs & Learning Curves

One full pass through the entire training set is one epoch. Training runs for many epochs, and plotting loss on the training and validation sets against epoch number is exactly ml1-8's own learning curve, applied here directly — the same diagnostic tool, the same reading (a persistent gap between the two curves signals overfitting; both curves converging to a poor value signals underfitting).

Learning Rate Schedules

nn1-5 already named the learning rate's own basic trade-off. In practice, many training runs use a schedule — starting with a larger learning rate for fast early progress, then gradually shrinking it as training proceeds, allowing the network to settle into a good solution more precisely once it's already in the right general area.

Why Neural Networks Overfit — ml1-8's Own Framing, New Model Family

A network with enough layers and neurons has an enormous number of learnable parameters — genuinely enough capacity to memorize a training set outright, exactly the same failure mode ml1-7's own unconstrained decision tree demonstrated most visibly. ml1-8's own bias-variance tradeoff applies unchanged: more capacity can mean lower bias and higher variance, and every technique below is a deliberate, controlled trade of one against the other, for this specific model family.

Dropout — A Genuine Cousin of ml1-7's Own Random Forest

Dropout randomly "turns off" (zeros out) a fraction of a layer's own neurons on each individual training pass — a different random subset every time. No single neuron can be relied upon to always be present, so the network is forced to spread useful information across many redundant pathways rather than concentrating it in a few neurons that happen to fit training-set quirks especially well.

Early Stopping — Catching Overfitting Live

ml1-8's own overfitting signature — low training error, rising validation error — doesn't have to be diagnosed only after the fact. Early stopping monitors ml1-2's own validation set during training itself and halts once validation loss starts climbing even as training loss keeps falling, keeping the weights from the point right before overfitting began rather than the final, most-overfit epoch.

Combining the Toolkit

Hands-On Exercises

Every network so far has been fully connected — every neuron sees every input. This chapter covers why that breaks down for images, and delivers historyai3-4's own AlexNet story with the actual technical substance that narrative left out.

Why a Plain Fully-Connected Network Struggles With Images

A modest 224×224 color image already has 224 × 224 × 3 ≈ 150,000 input values. A fully-connected first hidden layer, even a modest one, would need roughly 150,000 weights per neuron — enormous, expensive, and prone to exactly nn1-6's own overfitting story given typical training-set sizes. Worse: a fully-connected layer treats every pixel independently, with no built-in notion that nearby pixels are related. Shift an object slightly in the frame, and the network has to essentially relearn the same pattern all over again at the new position — no built-in translation invariance.

Convolution & Kernels — Solving Both Problems at Once

A kernel (a small grid of learnable weights, commonly 3×3 or 5×5) slides across the image, computing a small, local weighted sum at each position. Crucially, the same small set of weights is reused at every position — parameter sharing.

Feature Maps — Many Kernels, Many Views

Applying one kernel across the whole image produces one feature map — a grid showing where that kernel's own pattern was detected, and how strongly. A real convolutional layer runs many kernels in parallel — one might learn to detect vertical edges, another horizontal edges, another a particular color transition or texture — each producing its own feature map, together giving the network many simultaneous "views" of the same image.

Pooling — Downsampling for Efficiency and Further Invariance

Max pooling shrinks a feature map by taking the maximum value within each small region (commonly 2×2), reducing the spatial size passed to later layers. This cuts computation for everything downstream, and adds a further degree of translation invariance — a small shift in exactly where a feature sits within its own pooling region no longer changes the output at all.

Stacking Layers — From Edges to Objects

Exactly nn1-3's own "depth composes transformations" principle, now made concrete visually: early convolutional layers learn simple features (edges, color blobs); later layers combine those into increasingly complex, abstract features (textures, shapes, eventually recognizable object parts and whole objects) — a genuine, hierarchical feature-learning story built entirely from stacking the same basic operation.

Delivering historyai3-4's Own AlexNet Story, Technically

An Honest Nod to imgai1-2's Own Diffusion U-Net

imgai1-2's own diffusion explainer described a denoising network without detailing its internal architecture. Worth naming honestly here, without overclaiming: the U-Net architecture commonly used inside a diffusion model's own denoising network is built from the same convolutional building blocks this chapter covers — convolution, feature maps, downsampling — arranged into a specific downsample-then-upsample shape suited to producing a full image rather than a single classification. Not the identical architecture this chapter describes step by step; the same underlying convolutional vocabulary, applied to a different job.

What's Next

CNNs assume 2D spatial structure — nearby pixels matter, position within the frame is what matters. Text and time-series data have a different structural assumption entirely: order matters, not 2D position. nn1-8 covers the architecture family built specifically for that.

Hands-On Exercises

nn1-7 covered data where 2D position matters. This chapter covers data where order matters instead — text, time series, audio — and delivers the real, worked-out version of nn1-4's own vanishing-gradient cliffhanger, in exactly the setting that makes it most severe.

Why Neither Feedforward Networks Nor CNNs Naturally Handle Sequences

Every network so far processes one fixed-size input and produces one output, with no notion of "what came before." Predicting the next word in a sentence genuinely needs context from every preceding word, not just a fixed local window — a fundamentally different requirement from anything nn1-1–nn1-7 were built to handle.

The RNN Idea — A Hidden State Carried Through Time

A recurrent neural network maintains a hidden state — a vector updated at every step of the sequence. At each time step, the network combines the current input with the previous hidden state to compute a new one, which then carries forward to the next step. This hidden state functions as a compressed memory of everything the network has seen in the sequence so far.

Unrolling Through Time — Backpropagation Through Time

Conceptually "unrolling" an RNN means drawing the same cell repeatedly, once per time step, each copy feeding its own output forward as the next copy's own input. Training happens via backpropagation through time (BPTT) — literally nn1-5's own backpropagation and chain rule, applied across the unrolled time steps instead of across depth-wise layers.

Vanishing Gradients — nn1-4's Own Cliffhanger, Now Worse

nn1-4 previewed saturation shrinking gradients toward zero. In an RNN, this becomes genuinely more severe, for a specific, mechanical reason: because the same weight is reused at every time step, backpropagation through time multiplies that same weight's own gradient contribution by itself, repeatedly, once per time step. If each individual factor is even slightly less than 1 — easily the case with saturating activations — the cumulative product shrinks toward zero exponentially fast across a long sequence.

LSTMs — The Real, Documented Fix

Long Short-Term Memory networks introduce a separate cell state that flows through time with only carefully controlled modifications, plus three gates — small sigmoid-activated sub-networks that learn how much old information to forget, how much new information to add, and how much of the current state to actually output at each step.

What Comes Next — A Different Kind of Limit

LSTMs fixed the vanishing-gradient problem. They didn't fix a separate, genuinely different limitation: an RNN (LSTM or not) is inherently sequential — each time step's own computation depends on the previous step's own output, which means, unlike nn1-7's own CNN feature maps or nn1-6's own parallel mini-batches, the steps within one single sequence can't be computed simultaneously. nn1-9 covers the architecture that solved this specific bottleneck instead.

Hands-On Exercises

nn1-8 closed on a real, unsolved bottleneck: LSTMs fixed vanishing gradients but never fixed RNNs' own inherently sequential computation. This chapter previews the architecture that solved that specific problem — and, along the way, delivers part of historyai3-6's own "Attention Is All You Need" namecheck. The rest is llm1's own job.

The Motivating Question

Can a network capture relationships between distant elements in a sequence — nn1-8's own hidden/cell state's whole purpose — without forcing every step to wait for the one before it?

Self-Attention — A Direct, All-at-Once Relationship

Instead of carrying a compressed hidden state step by step, self-attention lets every position in a sequence directly look at every other position simultaneously, computing a weighted combination based on how relevant each other position actually is. Nothing is mediated through a chain of intermediate steps — the relationship between position 1 and position 50 is computed exactly as directly as the relationship between two adjacent positions.

Solving Both of nn1-8's Own Open Problems

The long-range-dependency fix is genuinely structural, not a patch layered on top the way nn1-8's own LSTM gating was — there's no long chain for a gradient to vanish across in the first place, because the relationship is computed directly rather than relayed through many intermediate steps. The parallelization fix directly resolves nn1-8's own closing cliffhanger, and aligns naturally with nn1-6's own GPU-parallel training.

A Deliberately Honest Gap — Order

Self-attention, by itself, has no inherent notion of sequence order at all — computing how much "it" attends to "cat" doesn't naturally encode whether "cat" came before or after "it" in the sentence. Unlike an RNN, which processes tokens strictly in order by construction, a transformer needs a separate, explicit mechanism to inject position information back in — positional encoding, one of several real technical pieces this preview deliberately leaves for llm1 to cover in full.

What's Deliberately Deferred to llm1

Beyond Language

The same self-attention idea generalized well past its original text-focused motivation — vision transformers apply the identical mechanism to image patches instead of word tokens, a real, documented extension of the architecture beyond the problem it was originally built to solve. Not this course's own focus; worth knowing the architecture didn't stay confined to text.

Hands-On Exercises

nn1-1–nn1-9 built the concepts. This chapter maps them onto real, working code — and covers the two dominant frameworks the rest of this course's own ecosystem (llm1 included) is built on.

Two Frameworks, Two Different Starting Philosophies

TensorFlow (Google) originally used define-then-run: build the entire computation graph first, as a fixed structure, then feed data through it. Efficient for deployment and optimization, but genuinely awkward for debugging — you couldn't simply inspect an intermediate value mid-computation the way you'd step through ordinary Python. PyTorch (Meta) used eager execution — define-by-run — from the start: the computation graph is built dynamically as code actually executes, line by line, exactly like ordinary Python. This genuinely mattered for research and experimentation, and is a real, documented reason PyTorch became — and largely remains — the dominant framework in academic and research settings specifically.

An Honest Update — This Gap Has Narrowed

The Current Practical Landscape

A Real Small Network — Every Concept, Real Code

import torch
import torch.nn as nn
import torch.optim as optim

class SmallNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(2, 4)   # nn1-1's own hidden layer
        self.output = nn.Linear(4, 1)   # nn1-1's own output layer
        self.relu = nn.ReLU()           # nn1-4's own activation function

    def forward(self, x):
        x = self.relu(self.hidden(x))   # nn1-1's own forward propagation
        return torch.sigmoid(self.output(x))   # ml1-5's own sigmoid, one more time

model = SmallNetwork()
loss_fn = nn.BCELoss()                          # cross-entropy for binary output (nn1-5)
optimizer = optim.SGD(model.parameters(), lr=0.1)   # nn1-6's own gradient descent, with a learning rate

for epoch in range(1000):                       # nn1-6's own epochs
    predictions = model(X_train)                # forward pass
    loss = loss_fn(predictions, y_train)
    optimizer.zero_grad()
    loss.backward()                              # nn1-5's own backpropagation, one line
    optimizer.step()                             # the weight update itself

Hands-On Exercises

One real dataset, already familiar: ds1-10's own employee-attrition table, already fit with ml1-5's logistic regression and ml1-7's random forest. This capstone builds a real, working neural network on the exact same problem — and asks the honest question this course owes a real answer to: does going deeper actually help here?

The Full Pipeline

import torch
import torch.nn as nn
import torch.optim as optim

X = pd.get_dummies(df[["department", "age", "years_at_company", "salary"]])
y = df["left_company"].map({"Yes": 1, "No": 0})
# train/val/test split — ml1-2, nn1-6
# StandardScaler — ds1-2/ml1-3

class AttritionNet(nn.Module):
    def __init__(self, n_features):
        super().__init__()
        self.hidden1 = nn.Linear(n_features, 16)
        self.hidden2 = nn.Linear(16, 8)
        self.dropout = nn.Dropout(0.3)     # nn1-6
        self.output = nn.Linear(8, 1)
        self.relu = nn.ReLU()              # nn1-4

    def forward(self, x):
        x = self.relu(self.hidden1(x))
        x = self.dropout(x)
        x = self.relu(self.hidden2(x))
        return torch.sigmoid(self.output(x))   # nn1-1's own closing callback — see below

model = AttritionNet(n_features=X.shape[1])
loss_fn = nn.BCELoss()                                    # nn1-5
optimizer = optim.Adam(model.parameters(), lr=0.01)        # nn1-6

best_val_loss = float("inf")
for epoch in range(200):
    # forward pass, loss, backward(), step() — nn1-5/nn1-6
    # track train_loss and val_loss each epoch — nn1-6's own learning curve
    # early stopping — nn1-6: save weights when val_loss improves

Evaluating — The Same Metrics, For a Fair Comparison

from sklearn.metrics import precision_score, recall_score, f1_score
preds = (model(X_test) > 0.5).int()   # ml1-6's own threshold, applied here too
precision_score(y_test, preds), recall_score(y_test, preds), f1_score(y_test, preds)

ml1-6's own precision/recall/F1 apply completely unchanged — the same evaluation vocabulary works identically regardless of which model produced the predictions.

The Honest Finding

Chapter Attribution

Honest Scope Note

Hands-On Exercises