ds1 taught data handling, ml1 taught classical models, nn1 taught neural architectures — all three, so far, applied to numbers and categories. This course applies all of it to text, a genuinely messier, more ambiguous kind of data than anything the first three courses ever handed you.

Why Text Is ds1-4's Own Data-Cleaning Material, Taken to Its Extreme

ds1-4 covered missing values, duplicates, inconsistent formatting, and string cleaning — real problems, but confined to a handful of structured columns with a known, fixed shape. A single paragraph of raw text has every one of those same problems and none of the fixed structure: no defined length, no fixed vocabulary, genuine ambiguity (the same word meaning different things in different contexts), and grammar and word order that carry real information a plain column of numbers never has to represent at all.

The Text Preprocessing Pipeline

Tokenization

"I don't think it's working." → ["I", "do", "n't", "think", "it", "'s", "working", "."]

Splitting text into tokens — usually words, sometimes subwords or characters — is the first, foundational step everything else builds on.

Lowercasing

Treating "The" and "the" as the same token reduces vocabulary size and noise — genuinely useful for most tasks, and a real, honest trade-off worth naming: "US" (the country) and "us" (the pronoun) collapse into the same token once lowercased, quietly discarding a real distinction some tasks would actually need.

Stopword Removal

Words like "the", "a", "is" appear constantly and usually carry little distinguishing information — removing them reduces noise for many tasks.

Stemming vs. Lemmatization

What This Pipeline Doesn't Do Yet

After all four steps, you have a cleaner set of tokens — still just words, not numbers. Every model this course eventually uses, from ml1's own classifiers to nn1's own networks, needs numeric input. nlp1-2 covers the first real way to get there.

This Course's Own Roadmap

Hands-On Exercises

nlp1-1 left off with clean tokens — still just words. This chapter turns them into the one thing every model since ml1-3 has actually required: numbers.

The Bag-of-Words Idea

Build a vocabulary — every unique word across the whole corpus. Represent each document as a vector with one position per vocabulary word, where the value is how many times that word appears in that document. "Bag" because order is discarded entirely — as if every word were tipped out of the document into an unordered bag and only the counts were kept.

Doc 1: "free money now"
Doc 2: "call me now"

Vocabulary: [call, free, me, money, now]

Doc 1 vector: [0, 1, 0, 1, 1]
Doc 2 vector: [1, 0, 1, 0, 1]

Feeding This Directly Into ml1-5's Own Logistic Regression

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)   # exactly the "X" ml1-3/ml1-5 already used
y = labels                                # spam / not spam

model = LogisticRegression()
model.fit(X, y)                            # ml1-5's own workflow, unchanged

Two Real, Practical Details

Sparse Vectors

A real corpus can easily have a vocabulary of tens of thousands of unique words — but any single document only ever uses a tiny fraction of them. The resulting vectors are overwhelmingly zeros, a property called sparsity. This is a genuinely different practical concern from ds1's own typically dense numeric tables, though most libraries (including CountVectorizer) store and compute with sparse vectors efficiently rather than wastefully storing every zero.

Words Never Seen During Training

The vocabulary is built from the training data. A brand-new document at prediction time containing a word the vocabulary has never seen simply has nowhere to put it — that word is typically dropped entirely, silently. A real, honest limitation worth naming plainly.

What This Chapter Deliberately Doesn't Fix

Hands-On Exercises

nlp1-2's own raw counts treat every word's occurrence as equally meaningful. It isn't — a word appearing constantly across every document tells you almost nothing about what makes any one of them distinctive.

Why Raw Counts Alone Are Misleading

A word like "now" might appear often in a spam message — and just as often in an ordinary one. Its raw count is high, but it does little to actually distinguish spam from not-spam. A genuinely rare word like "viagra", by contrast, might appear only once — but that single occurrence is far more informative about what kind of message this is.

TF — Term Frequency

How often a word appears within this document, typically normalized by the document's own total word count — ds1-6's own proportion vocabulary, applied directly: not a raw count, but a rate, so a short and a long document can be compared fairly.

TF(word, doc) = (count of word in doc) / (total words in doc)

IDF — Inverse Document Frequency

How rare a word is across the whole corpus. A word appearing in nearly every document gets a low IDF (not distinctive); a word appearing in only a handful gets a high IDF (distinctive).

IDF(word) = log( total documents / documents containing word )

The logarithm is a deliberate dampener — without it, an extremely rare word's own score could grow disproportionately large and dominate everything else; the log keeps the scale reasonable while still rewarding rarity.

TF-IDF = TF × IDF

A word scores high when it's frequent in this document and rare across the corpus overall — genuinely distinctive for this specific document. It scores low when it's either rare here or common everywhere.

In Practice — A Drop-In Replacement

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)   # same shape, same downstream ml1-5 workflow

TfidfVectorizer slots directly into nlp1-2's own pipeline in place of CountVectorizer — everything downstream, including ml1-5's own LogisticRegression.fit(), is unchanged.

What TF-IDF Does Not Fix

Hands-On Exercises

nlp1-2 and nlp1-3 both flagged the same loss and moved on. This chapter stops and actually proves it, then reveals a second, entirely separate blind spot neither chapter mentioned at all.

Problem 1: Word Order — Proven, Not Just Asserted

Vocabulary: [bites, dog, man].

"dog bites man" → [1, 1, 1]
"man bites dog" → [1, 1, 1]

Identical vectors. Not approximately similar — identical, down to the last digit, whether built with nlp1-2's own raw counts or nlp1-3's own TF-IDF weights. A model trained on either representation has no way whatsoever to distinguish these two sentences, because as far as its own input is concerned, they aren't two sentences at all — they're the exact same input, twice.

Problem 2: No Notion of Meaning — A Genuinely Separate Issue

Every vocabulary word is its own independent, arbitrary dimension. "good" and "great" are just two unrelated vector positions — as far as the representation itself is concerned, "good" is exactly as similar to "great" as it is to "car" or "purple". Nothing about bag-of-words or TF-IDF encodes synonymy or semantic closeness at all.

This also reframes nlp1-2's own out-of-vocabulary limitation as a direct symptom of this deeper problem, not just an annoying edge case: a genuinely new word like "excellent" gets silently dropped rather than recognized as close in meaning to "great," precisely because nothing in the representation has any concept of "close in meaning" to begin with.

Two Separate Problems, Two Separate Fixes

Hands-On Exercises

nlp1-4 diagnosed Problem 2: every word is its own isolated, unrelated dimension. This chapter fixes it — and reveals that the tool doing the fixing is something you already know how to build.

Word Embeddings — The Core Idea

Instead of a huge, sparse vector with one arbitrary dimension per vocabulary word, represent each word as a small, dense vector — commonly 100–300 numbers — learned so that semantically similar words end up genuinely close together in that vector space. nlp1-4's own example, resolved directly: in embedding space, "good" and "great" sit near each other; "good" and "car" sit far apart. A real, learned, geometrically meaningful relationship — not an arbitrary one.

Word2Vec — Learning This From Raw Text Alone

Word2vec learns embeddings self-supervised — no manual labeling at all, purely from patterns in how words co-occur across a huge corpus of ordinary, unlabeled text. Two real training setups:

Word2Vec Is Literally nn1-1's Own Neuron, Generalized

Word2vec's own architecture is a small feedforward network, built entirely from nn1-1's own vocabulary: an input layer (a one-hot encoded word), a single hidden layer, and an output layer predicting the context or target word.

The Geometric Payoff — Honestly Framed

Trained on a large enough corpus, the resulting embedding space captures genuine semantic and syntactic relationships well enough that vector arithmetic becomes meaningful: the famous king − man + woman ≈ queen.

In Practice — Pretrained, Not Trained From Scratch

Training word2vec from scratch needs a genuinely large corpus. In practice, pretrained embeddings — trained once, on massive text collections, by someone else — are typically used directly rather than retrained per project. nlp1-9 covers this properly; for now, know that it's the normal way embeddings actually get used.

What's Still Broken

nlp1-4's meaning problem is solved. Its other diagnosed problem — word order — is completely untouched by anything in this chapter. Averaging or otherwise combining word embeddings without regard to their sequence still loses order entirely. nlp1-6 is where that gets fixed.

Hands-On Exercises

nlp1-5 fixed meaning. This chapter fixes the other half of nlp1-4's own diagnosis — word order — by finally putting nn1-8's own RNN/LSTM material to real, concrete use on real text.

The Real Pipeline, End to End

Resolving nlp1-4's Own Cliffhanger, Concretely

nn1-8's own hidden-state update at every step depends on both the current input and the previous hidden state — order-dependent by construction. Feed the embeddings for "dog", "bites", "man" into the LSTM in that exact order, and the resulting final hidden state is genuinely, mathematically different from feeding "man", "bites", "dog" — because at every intermediate step, the hidden state carries forward a different history depending on what came before it.

A Real Worked Task — Sentiment Classification

import torch.nn as nn

class TextClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)   # nlp1-5's own embeddings
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)   # nn1-8
        self.output = nn.Linear(hidden_dim, 1)                  # nn1-1's own closing callback, again

    def forward(self, x):
        embedded = self.embedding(x)              # one embedding per token, in order
        _, (hidden, _) = self.lstm(embedded)        # the final hidden state — nn1-8's own "memory" of the whole sequence
        return torch.sigmoid(self.output(hidden[-1]))   # ml1-5's own logistic regression, one more time

Both Problems, Genuinely Solved Together

nlp1-4 diagnosed two independent problems — meaning and order — and insisted neither fix alone solves both. This chapter's own pipeline is the actual combination: nlp1-5's embeddings supply meaning, this chapter's own sequence model supplies order. Only together do they produce a representation that genuinely distinguishes "dog bites man" from "man bites dog" and recognizes "excellent" as similar to "great."

What's Next

This chapter's own model produces exactly one output for an entire sequence — a single sentiment label per document. nlp1-7 extends the identical underlying idea to produce one output per token instead — real, practical tasks like recognizing names and grammatical roles word by word.

Hands-On Exercises

nlp1-6's own model produced exactly one output for an entire sequence. This chapter needs one output per token instead — and the architectural change required is smaller than it sounds.

Two Real, Practical Tasks

Named Entity Recognition (NER) — identifying and classifying named entities in text:

Part-of-Speech (POS) tagging — assigning each word its grammatical role:

Both are sequence labeling tasks — a genuine, distinct category from nlp1-6's own sequence classification: one label for every token, not one label for the whole sequence.

The Architectural Change — Smaller Than It Sounds

nlp1-6's own model discarded every hidden state except the very last one, using it as a single summary of the entire sequence. Sequence labeling keeps every hidden state — one per time step — and feeds each one individually through the same small classifier head, producing one prediction per token instead of one prediction total.

class SequenceLabeler(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_tags):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True, bidirectional=True)
        self.output = nn.Linear(hidden_dim * 2, num_tags)

    def forward(self, x):
        embedded = self.embedding(x)
        outputs, _ = self.lstm(embedded)     # every hidden state, not just the last (nlp1-6's own change)
        return self.output(outputs)           # one prediction per token, via softmax over num_tags

Bidirectional LSTMs — Why "Later" Words Help Too

"Washington" could be a person's surname or a place — often only resolvable by what comes after it in the sentence, not before. nn1-8's own plain LSTM only ever sees what came earlier. A bidirectional LSTM (BiLSTM) runs two LSTMs over the same sequence — one forward, one backward — and concatenates both hidden states at every position, so each token's own final representation genuinely reflects the whole sentence, not just its own left-hand context.

Hands-On Exercises

nn1-9 previewed self-attention conceptually and deliberately deferred full depth to llm1. This chapter doesn't skip ahead of that plan — it grounds the preview in the real, historically accurate problem attention was invented to solve, which happens to be an NLP problem: machine translation.

Seq2Seq: nlp1-6's Own Architecture, Turned Toward Generation

Translating a sentence is naturally framed as sequence-to-sequence: an encoder LSTM reads the entire source sentence, and a decoder LSTM generates the target sentence word by word. Look closely at the encoder's own job, though — it is doing exactly what nlp1-6's classifier did: processing a whole sequence and compressing it down into a single final hidden state. The only difference here is what happens to that summary afterward — instead of feeding it to a sigmoid, it's handed to a second LSTM as its own starting point.

The Fixed-Vector Bottleneck

For a short sentence, squeezing everything into one vector works reasonably well. For a long sentence, it's a genuine problem: the encoder's final hidden state has to somehow carry everything the decoder will need — every noun, every clause, every dependency — through one fixed-size vector. Early words in a long sentence get progressively diluted by every step that comes after them, the same accumulating-history mechanism nlp1-6 relied on to preserve order now works against fidelity at length.

Attention: Let the Decoder Look Back

Bahdanau's 2014 fix (pre-dating the 2017 Transformer paper by three years) was simple in concept: instead of forcing the decoder to work from one final encoder hidden state, let it look back at every encoder hidden state at every step it generates, and compute a weighted combination — attending more heavily to whichever source words are most relevant to the word being generated right now.

# conceptual attention score at one decoder step
scores = [dot(decoder_state, encoder_state_i) for encoder_state_i in encoder_states]
weights = softmax(scores)                     # how much to "attend" to each source word
context = sum(w * s for w, s in zip(weights, encoder_states))   # weighted blend, not one fixed vector

Translating "the black cat" into French, the decoder generating "noir" (black) would learn to assign a high attention weight to the source word "black" and comparatively little to "the" or "cat" — a direct, interpretable link between output and source, something the single fixed vector could never expose.

Self-Attention: The 2017 Leap

The real leap in "Attention Is All You Need" (2017) — the paper historyai3-6 already namechecked — was applying this exact same mechanism within a single sequence, not just from decoder back to encoder. In self-attention, every token computes attention weights against every other token in the same sequence, building a representation of each word informed by every other word it's relevant to — no recurrence required at all.

What's Still Deferred

nn1-9 already flagged this honestly, and it's still true here: self-attention has no inherent sense of token order — swap two tokens and the set of pairwise relationships computed is identical. A real Transformer needs positional encoding to reintroduce order information, and the full multi-head, multi-layer Transformer architecture built from self-attention is genuinely substantial. Both are deliberately left for llm1's own full depth, exactly as planned.

Hands-On Exercises

nlp1-5 trained word2vec embeddings from scratch, on whatever corpus happened to be on hand. This chapter asks the obvious next question: what if, instead, someone else already trained embeddings on a corpus vastly larger than any single project could gather — and all that's needed is to reuse them?

The Cold-Start Problem nlp1-5 Quietly Assumed Away

Word2vec's own skip-gram/CBOW training (nlp1-5) needs a large corpus and real training time to learn good relationships — "excellent" only ends up near "great" in vector space after seeing both words in enough similar contexts, many times over. A small project's own dataset — the spam examples from nlp1-2, or the toy sentences carried through nlp1-6/nlp1-7 — is nowhere near large enough to learn quality embeddings from scratch. Training on too little text produces noisy, unreliable vectors no matter how correct the algorithm is.

GloVe — A Genuinely Different Training Approach

GloVe (Global Vectors, Stanford) is the other major pretrained-embedding family alongside word2vec, and it gets there differently. Word2vec learns from local context windows, one sliding prediction task at a time. GloVe instead builds a full word-by-word co-occurrence matrix across an entire corpus (how often does word A appear near word B, counted globally) and factorizes that matrix directly to produce vectors — a genuinely different mathematical approach reaching a similar kind of representation. Trained on massive corpora (Wikipedia, Common Crawl — billions of words), the resulting vectors are freely downloadable and ready to use without any training of your own.

Transfer Learning — A Pattern, Not Just an NLP Trick

Reusing embeddings trained on one (huge, general) task for a different (smaller, specific) task is an instance of transfer learning — a general pattern, not unique to text. nn1-7's own AlexNet, trained on ImageNet's millions of labeled images, produces internal features so generally useful that they get reused as a starting point for entirely different image tasks the network was never trained on. Pretrained word embeddings are the same idea in a different modality: a representation learned once, on a huge general corpus, reused as a head start rather than relearned from nothing.

Frozen vs. Fine-Tuned

There are two honest ways to use a pretrained embedding layer:

# frozen — the classic transfer-learning move for a small dataset
embedding = nn.Embedding.from_pretrained(glove_vectors, freeze=True)

# fine-tuned — start from GloVe, keep adapting during training
embedding = nn.Embedding.from_pretrained(glove_vectors, freeze=False)

Compare this to nlp1-5's own nn.Embedding(vocab_size, embed_dim) — randomly initialized, learned entirely from this project's own limited data. Loading GloVe's own pretrained values instead means training starts from vectors that already encode real semantic relationships, rather than from random noise.

The Honest Limitation: Out-of-Vocabulary Words

The Bridge Into llm1

Hands-On Exercises

Nine chapters, each solving one specific, honestly-diagnosed problem. This capstone assembles the pieces that actually survive into a real, working sentiment-classification pipeline — then closes with the question this whole course has been quietly building toward: what does an LLM actually do differently?

The Pipeline

class SentimentClassifier(nn.Module):
    def __init__(self, glove_vectors, hidden_dim):
        super().__init__()
        # nlp1-9: pretrained GloVe, frozen — this dataset is small, per nlp1-9's own guidance
        self.embedding = nn.Embedding.from_pretrained(glove_vectors, freeze=True)
        # nlp1-6: LSTM sequence model, resolving nlp1-4's order-and-meaning problem together
        self.lstm = nn.LSTM(glove_vectors.shape[1], hidden_dim, batch_first=True)
        # nlp1-6: classifier head reading only the final hidden state — one label per sequence
        self.output = nn.Linear(hidden_dim, 1)

    def forward(self, x):
        embedded = self.embedding(x)              # nlp1-1: x is already tokenized/cleaned text
        _, (hidden, _) = self.lstm(embedded)
        return torch.sigmoid(self.output(hidden[-1]))   # binary sentiment, per nlp1-6

Every line here has a specific origin. Text arrives already tokenized and cleaned (nlp1-1), is converted to vectors using pretrained, frozen GloVe embeddings rather than training from scratch on this project's own small dataset (nlp1-9, resolving the exact cold-start problem that chapter diagnosed), processed in order by an LSTM so word order genuinely affects the result (nlp1-6, resolving nlp1-4's own two-problem cliffhanger together with the embeddings), and classified from the final hidden state through a single sigmoid output (nlp1-6's own binary classifier head).

Chapter Attribution Table

Why LLMs Are Different

Look honestly at what this course actually built. nlp1-6's sentiment classifier and nlp1-7's NER tagger share the same underlying LSTM mechanism — but they are genuinely different pipelines: different output layers, different training objectives, separately trained models. Every task this course covered got its own purpose-built, hand-assembled architecture. That's not a flaw in how the course was taught — it's an honest reflection of how NLP actually worked before LLMs.

Put plainly: this course taught how to hand-build the right tool for each job. An LLM is closer to one very large, very capable tool that has already seen enough language to be pointed at most jobs without being rebuilt from scratch each time. Neither approach makes the other obsolete to understand — knowing exactly what a purpose-built pipeline is doing, and why, is exactly what makes it possible to reason honestly about what a much larger, more opaque model might be doing instead.

Hands-On Exercises

Scope Note — What This Capstone Deliberately Doesn't Do