What We're Building

The identical architecture nlp1-10 already built and proved works: preprocessing (nlp1-1) → frozen pretrained GloVe embeddings (nlp1-9) → an LSTM sequence classifier (nlp1-6). Nothing about the model changes here — what changes is scale: a real, freely available, canonical sentiment dataset instead of a toy example, and everything that comes with training on tens of thousands of real examples instead of a handful.

Step 1: Loading the Real Dataset

import pandas as pd

df = pd.read_csv("imdb_reviews.csv")   # columns: review, sentiment ("positive"/"negative")
print(df.shape)
print(df["sentiment"].value_counts())

The IMDb movie review dataset is the canonical benchmark for exactly this task — 25,000 real reviews, roughly balanced between positive and negative, freely available and genuinely unremarkable to work with (no scraping, no missing values, no cleaning pipeline needed here — that work belongs to Chapter 2's own dataset, not this one).

Step 2: Preprocessing — nlp1-1's Own Pipeline, Reused Directly

import re

def preprocess(text):
    text = text.lower()
    text = re.sub(r"[^a-z\s]", "", text)
    return text.split()

df["tokens"] = df["review"].apply(preprocess)

nlp1-1's own tokenization/cleaning approach, applied unchanged — nothing here needed to be redesigned for real data, since preprocessing was always meant to generalize beyond whatever toy examples originally demonstrated it.

Step 3: Building a Real Vocabulary

from collections import Counter

all_tokens = [tok for tokens in df["tokens"] for tok in tokens]
vocab = {word: i + 2 for i, (word, count) in enumerate(Counter(all_tokens).most_common(20000))}
vocab["<pad>"] = 0
vocab["<unk>"] = 1
print(f"Vocabulary size: {len(vocab)}")

Step 4: Loading Only the GloVe Vectors This Vocabulary Actually Needs

import numpy as np

embedding_dim = 100
embedding_matrix = np.zeros((len(vocab), embedding_dim))

with open("glove.6B.100d.txt", encoding="utf-8") as f:
    for line in f:
        parts = line.split()
        word = parts[0]
        if word in vocab:
            embedding_matrix[vocab[word]] = np.array(parts[1:], dtype="float32")

The full GloVe file (nlp1-9's own pretrained embeddings) contains vectors for roughly 400,000 words — loading every single one into memory just to use 20,000 of them would be genuine, avoidable waste. This loop reads the file once and keeps only the rows this specific vocabulary needs.

Step 5: The Exact Model From nlp1-6/nlp1-10 — Unchanged

import torch
import torch.nn as nn

class SentimentClassifier(nn.Module):
    def __init__(self, embedding_matrix, hidden_dim=128):
        super().__init__()
        self.embedding = nn.Embedding.from_pretrained(
            torch.tensor(embedding_matrix, dtype=torch.float32), freeze=True
        )
        self.lstm = nn.LSTM(embedding_matrix.shape[1], hidden_dim, batch_first=True)
        self.output = nn.Linear(hidden_dim, 1)

    def forward(self, x):
        embedded = self.embedding(x)
        _, (hidden, _) = self.lstm(embedded)
        return torch.sigmoid(self.output(hidden[-1]))

Step 6: Training — Now With Real Batching

from torch.utils.data import Dataset, DataLoader

def encode(tokens, max_len=200):
    ids = [vocab.get(t, 1) for t in tokens[:max_len]]
    return ids + [0] * (max_len - len(ids))

class ReviewDataset(Dataset):
    def __init__(self, df):
        self.X = [encode(t) for t in df["tokens"]]
        self.y = (df["sentiment"] == "positive").astype("float32").values

    def __len__(self):
        return len(self.X)

    def __getitem__(self, i):
        return torch.tensor(self.X[i]), torch.tensor(self.y[i])

loader = DataLoader(ReviewDataset(df), batch_size=64, shuffle=True)

model = SentimentClassifier(embedding_matrix)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.BCELoss()

for epoch in range(3):
    for X_batch, y_batch in loader:
        optimizer.zero_grad()
        preds = model(X_batch).squeeze()
        loss = loss_fn(preds, y_batch)
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch+1} done.")

Step 7: Evaluating on Real Reviews

sample_reviews = [
    "This movie was absolutely fantastic, best film I've seen all year.",
    "Waste of two hours. Terrible acting, worse plot.",
]
for review in sample_reviews:
    encoded = torch.tensor([encode(preprocess(review))])
    score = model(encoded).item()
    print(f"{score:.2f} — {review[:50]}...")

A genuinely satisfying moment this project earns that a toy example can't: real, freshly-written sentences, never seen during training, scored by a model trained on real human-written reviews rather than a handful of illustrative examples.

What's Next

Chapter 2: Customer Churn Predictor — the full ml1 depth Chapter 7 of the Beginner course deliberately declined to use: real feature engineering, a genuine model comparison, cross-validation, and the full precision/recall/F1 picture.

What We're Building

A churn predictor for a telecom-style customer dataset — real subscriber data (tenure, monthly charges, contract type, services subscribed) used to predict whether a customer will cancel. Two real models, compared honestly: logistic regression (ml1-5) and a random forest (ml1-7), evaluated with the full metric picture ml1-6 covers, validated with real cross-validation (ml1-8).

Step 1: Real Feature Engineering

import pandas as pd

df = pd.read_csv("telco_churn.csv")

df["avg_monthly_spend"] = df["TotalCharges"] / df["tenure"].replace(0, 1)
df["is_long_term"] = (df["tenure"] > 24).astype(int)

categorical_cols = ["Contract", "InternetService", "TechSupport"]
df = pd.get_dummies(df, columns=categorical_cols, drop_first=True)

y = (df["Churn"] == "Yes").astype(int)
X = df.drop(columns=["customerID", "Churn", "TotalCharges"])

Genuinely new territory beyond dsproj1-7's own four ready-made numeric measurements: avg_monthly_spend and is_long_term are derived features, engineered because raw tenure and TotalCharges alone don't directly capture them. pd.get_dummies() one-hot encodes categorical columns like Contract into numeric 0/1 columns a model can actually use — real tabular data is rarely all-numeric the way Iris was.

Step 2: Split, and Why One Split Isn't Enough

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

stratify=y keeps the churned/not-churned ratio consistent between train and test sets — worth doing explicitly here, since churn datasets are rarely 50/50, exactly the situation Step 5 comes back to directly.

Step 3: Model 1 — Logistic Regression

from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression(max_iter=1000)
logreg.fit(X_train, y_train)

ml1-5's own model, applied here for real — a coefficient per feature, each one directly readable as "this feature pushes the prediction toward churn or away from it."

Step 4: Model 2 — Random Forest

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=100, random_state=42)
forest.fit(X_train, y_train)

ml1-7's own ensemble model, fit on the identical training data — an intentional apples-to-apples comparison against logistic regression.

Step 5: The Full Evaluation Picture

from sklearn.metrics import classification_report, confusion_matrix

for name, model in [("Logistic Regression", logreg), ("Random Forest", forest)]:
    preds = model.predict(X_test)
    print(f"\n--- {name} ---")
    print(classification_report(y_test, preds, target_names=["Stayed", "Churned"]))
    print(confusion_matrix(y_test, preds))

Step 6: Cross-Validation — A More Honest Estimate

from sklearn.model_selection import cross_val_score

for name, model in [("Logistic Regression", logreg), ("Random Forest", forest)]:
    scores = cross_val_score(model, X, y, cv=5, scoring="f1")
    print(f"{name}: F1 = {scores.mean():.3f} (+/- {scores.std():.3f})")

Per ml1-8, a single train/test split can be a lucky or unlucky draw — cv=5 fits and evaluates each model five separate times on five different splits, reporting the average and spread rather than trusting one number. The +/- spread itself is worth reading: a model whose F1 barely moves across folds is more trustworthy than one that swings wildly from one split to the next.

Step 7: Which Features Actually Drive Churn

importances = pd.Series(forest.feature_importances_, index=X.columns).sort_values(ascending=False)
print(importances.head(10))

A genuine business payoff beyond "the model works": feature_importances_ reveals which factors the random forest actually leaned on most — real, actionable information for whoever would act on these predictions, not just a black-box score.

What's Next

Chapter 3: Movie Recommender — returning to dsproj1-5's own ratings dataset, but with a genuinely different question: not what can be learned from the data, but what should be recommended from it.

What We're Building

A "users who liked this also liked..." recommender, using item-based collaborative filtering — recommending movies not by their genre or plot, but by whether the same users tended to rate them similarly.

Step 1: Loading & Merging — dsproj1-5's Own First Step

import pandas as pd

movies = pd.read_csv("movies.csv")
ratings = pd.read_csv("ratings.csv")
df = ratings.merge(movies, on="movie_id")

Step 2: Reshaping Into a User-Item Matrix

matrix = df.pivot_table(index="user_id", columns="title", values="rating")
print(matrix.shape)
print(matrix.iloc[:5, :5])

Step 3: An Honest Simplifying Assumption

matrix_filled = matrix.fillna(0)

Step 4: Cosine Similarity — nlp1-5's Own Idea, on Movies Instead of Words

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

similarity = cosine_similarity(matrix_filled.T)
similarity_df = pd.DataFrame(similarity, index=matrix_filled.columns, columns=matrix_filled.columns)

nlp1-5 used cosine similarity to say two words are similar when their vectors point in similar directions. Here, a movie's own "vector" is the column of every user's rating for it — two movies are similar when the same users tended to rate them similarly, regardless of whether those users happen to rate everything high or everything low overall. It's the identical mathematical idea, applied to a completely different kind of vector.

Step 5: Building the Recommender

def recommend(title, top_n=5):
    if title not in similarity_df.columns:
        return f"'{title}' not found in this dataset."
    scores = similarity_df[title].sort_values(ascending=False)
    return scores.iloc[1:top_n + 1]   # [0] is the movie itself, always similarity 1.0

scores.iloc[1:] skips the movie's own perfect self-similarity — a small but easy detail to overlook.

Step 6: Trying It on a Real Movie

print(recommend("The Matrix"))
# The Matrix Reloaded       0.71
# Terminator 2               0.64
# Inception                  0.61
# ...

Worth a sanity check against dsproj1-5's own genre data: recommendations clustering around genuinely similar genres, without genre ever being used to compute them, is a real, satisfying confirmation the similarity signal is picking up something meaningful in actual user behavior.

The Complete Recommender

import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

movies = pd.read_csv("movies.csv")
ratings = pd.read_csv("ratings.csv")
df = ratings.merge(movies, on="movie_id")

matrix = df.pivot_table(index="user_id", columns="title", values="rating").fillna(0)
similarity_df = pd.DataFrame(
    cosine_similarity(matrix.T), index=matrix.columns, columns=matrix.columns
)

def recommend(title, top_n=5):
    if title not in similarity_df.columns:
        return f"'{title}' not found."
    return similarity_df[title].sort_values(ascending=False).iloc[1:top_n + 1]

print(recommend("The Matrix"))

What's Next

Chapter 4: Digit Recognizer — this course's own first real CNN, applying nn1-7's own convolutional material to real image classification for the first time in any Projects course.

What We're Building

A digit recognizer for MNIST — 70,000 real handwritten digit images (0-9), the classic "hello world" of computer vision, comparable in stature to Iris for classification or IMDb for sentiment. It ships directly through torchvision, needs no cleaning, and is small enough to train in a reasonable time even without a GPU.

Step 1: Loading the Real Dataset

import torch
from torchvision import datasets, transforms

transform = transforms.ToTensor()
train_data = datasets.MNIST(root="data", train=True, download=True, transform=transform)
test_data = datasets.MNIST(root="data", train=False, download=True, transform=transform)

print(len(train_data), len(test_data))
print(train_data[0][0].shape)   # torch.Size([1, 28, 28]) — 1 grayscale channel, 28x28 pixels

Step 2: Looking at the Actual Images

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 5, figsize=(10, 2))
for i, ax in enumerate(axes):
    image, label = train_data[i]
    ax.imshow(image.squeeze(), cmap="gray")
    ax.set_title(str(label))
    ax.axis("off")
plt.savefig("sample_digits.png")

Worth seeing before building anything — these are genuinely messy, human-written digits, not clean printed text. A model that works here has to tolerate real handwriting variation, not just recognize one fixed font.

Step 3: The CNN — nn1-7's Own Architecture, Applied

import torch.nn as nn

class DigitCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2)
        self.fc = nn.Linear(32 * 7 * 7, 10)

    def forward(self, x):
        x = self.pool(torch.relu(self.conv1(x)))   # 28x28 -> 14x14
        x = self.pool(torch.relu(self.conv2(x)))   # 14x14 -> 7x7
        x = x.view(x.size(0), -1)             # flatten for the final linear layer
        return self.fc(x)

Per nn1-7: each Conv2d layer slides a small learned kernel across the image, extracting local features (edges, curves, strokes) rather than treating each pixel independently. MaxPool2d(2) halves the spatial size after each convolution, keeping the strongest signal from each small region — exactly the convolution-then-pooling pattern that chapter covered, applied here to real pixels instead of illustrative diagrams.

Step 4: Training — Batching, Reused From Chapter 1

from torch.utils.data import DataLoader

train_loader = DataLoader(train_data, batch_size=64, shuffle=True)
test_loader = DataLoader(test_data, batch_size=64)

model = DigitCNN()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(3):
    for images, labels in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(images), labels)
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch+1} done.")

The same DataLoader-based batching pattern dsproj2-1 already introduced — no new concept here, just applied to image tensors instead of encoded text sequences.

Step 5: Evaluating

correct, total = 0, 0
with torch.no_grad():
    for images, labels in test_loader:
        preds = model(images).argmax(dim=1)
        correct += (preds == labels).sum().item()
        total += labels.size(0)

print(f"Test accuracy: {correct / total:.2%}")   # typically well above 98%

Step 6: Looking at What It Actually Gets Wrong

wrong = []
with torch.no_grad():
    for images, labels in test_loader:
        preds = model(images).argmax(dim=1)
        mismatches = (preds != labels).nonzero()
        for idx in mismatches[:5]:
            i = idx.item()
            wrong.append((images[i], labels[i].item(), preds[i].item()))
        if wrong:
            break

fig, axes = plt.subplots(1, len(wrong), figsize=(10, 2))
for ax, (img, actual, predicted) in zip(axes, wrong):
    ax.imshow(img.squeeze(), cmap="gray")
    ax.set_title(f"actual {actual}, predicted {predicted}")
    ax.axis("off")
plt.savefig("misclassified.png")

What's Next

Chapter 5: Weather Forecaster — returning to dsproj1-3's own logged weather data, this time treating it as a genuinely different task shape: predicting tomorrow from today, not classifying independent rows.

What We're Building

A next-day temperature forecaster, built by turning a sequential problem into an ordinary regression problem through feature engineering — teaching a model to predict today's temperature from the last few days' own values, then reusing that same model to genuinely forecast tomorrow.

Step 1: Loading the Log, In Order

import pandas as pd

df = pd.read_csv("weather_log.csv", parse_dates=["time"])
df = df.sort_values("time").reset_index(drop=True)

Step 2: Resampling to Daily Granularity

daily = df.set_index("time").resample("D")["temperature"].mean().dropna().reset_index()

dsproj1-4's own .resample() tool, reused directly — dsproj1-3's own log is hourly; a next-day forecast is a daily-granularity problem.

Step 3: Engineering Lag Features

daily["temp_lag1"] = daily["temperature"].shift(1)
daily["temp_lag2"] = daily["temperature"].shift(2)
daily["temp_lag3"] = daily["temperature"].shift(3)

.shift(1) pulls each row's value from one position earlier into the current row — temp_lag1 on today's row literally is yesterday's temperature, now sitting as an input feature a regular regression model can use. This is the actual trick that turns a sequential forecasting problem into an ordinary tabular one.

Step 4: The Honest, Structural Missing Values This Creates

daily = daily.dropna().reset_index(drop=True)

The very first rows have no earlier days to pull a lag value from at all — temp_lag3 is undefined for day 1, 2, and 3. This is a structural byproduct of the lag-feature technique itself, not messy data the way dsproj1-2's own missing values were — nothing to fill or investigate, just a real, small, unavoidable cost of the method.

Step 5: The Split — Why Random Would Be Wrong Here

split_point = int(len(daily) * 0.8)
train = daily.iloc[:split_point]
test = daily.iloc[split_point:]

Step 6: Fitting the Model

from sklearn.linear_model import LinearRegression

features = ["temp_lag1", "temp_lag2", "temp_lag3"]
model = LinearRegression()
model.fit(train[features], train["temperature"])

ml1-3's own linear regression, applied here without modification — the model itself doesn't know or care that its inputs happen to be lagged values; that's entirely a property of how the features were engineered in Step 3.

Step 7: Evaluating — Against a Genuinely Honest Baseline

from sklearn.metrics import mean_absolute_error

predictions = model.predict(test[features])
model_mae = mean_absolute_error(test["temperature"], predictions)

naive_predictions = test["temp_lag1"]   # "tomorrow will be the same as today"
naive_mae = mean_absolute_error(test["temperature"], naive_predictions)

print(f"Model MAE: {model_mae:.2f}")
print(f"Naive baseline MAE: {naive_mae:.2f}")

The Complete Forecaster

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

df = pd.read_csv("weather_log.csv", parse_dates=["time"]).sort_values("time")
daily = df.set_index("time").resample("D")["temperature"].mean().dropna().reset_index()

for lag in (1, 2, 3):
    daily[f"temp_lag{lag}"] = daily["temperature"].shift(lag)
daily = daily.dropna().reset_index(drop=True)

split = int(len(daily) * 0.8)
train, test = daily.iloc[:split], daily.iloc[split:]
features = ["temp_lag1", "temp_lag2", "temp_lag3"]

model = LinearRegression().fit(train[features], train["temperature"])
model_mae = mean_absolute_error(test["temperature"], model.predict(test[features]))
naive_mae = mean_absolute_error(test["temperature"], test["temp_lag1"])

print(f"Model MAE: {model_mae:.2f} vs. naive baseline: {naive_mae:.2f}")

What's Next

Chapter 6: Named Entity Extractor — applying nlp1-7's own NER material to a real batch-document extraction tool, a structural sequel to Chapter 1's own whole-document sentiment classification.

What We're Building

An entity extractor identifying people, organizations, and locations across a real batch of documents — trained on CoNLL-2003, the canonical NER benchmark (comparable in stature to MNIST for images or IMDb for sentiment), then run over genuinely new text to produce structured, per-document extracted-entity output.

Step 1: A Real Labeling Scheme, More Detailed Than nlp1-7's Own

# CoNLL-2003 format: one token and tag per line
Sarah    B-PER
Chen     I-PER
works    O
at       O
New      B-ORG
York     I-ORG
Times    I-ORG

Step 2: Vocabulary — Reusing Chapter 1's Own Approach

from collections import Counter

all_tokens = [tok for sentence in sentences for tok, tag in sentence]
vocab = {word: i + 2 for i, (word, _) in enumerate(Counter(all_tokens).most_common(15000))}
vocab["<pad>"], vocab["<unk>"] = 0, 1

tag_set = sorted({tag for sentence in sentences for _, tag in sentence})
tag2idx = {tag: i for i, tag in enumerate(tag_set)}

Same frequency-capped vocabulary pattern dsproj2-1 established — plus a second small vocabulary, this time for the tag set itself.

Step 3: The Exact BiLSTM Tagger From nlp1-7

import torch.nn as nn

class SequenceLabeler(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_tags):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True, bidirectional=True)
        self.output = nn.Linear(hidden_dim * 2, num_tags)

    def forward(self, x):
        embedded = self.embedding(x)
        outputs, _ = self.lstm(embedded)
        return self.output(outputs)   # one prediction per token, per nlp1-7

Character for character nlp1-7's own architecture — every hidden state kept, bidirectional for exactly the "Washington could be a place or a person" reason that chapter demonstrated.

Step 4: Training — A Real Batching Gotcha Toy Data Never Surfaced

import torch

loss_fn = nn.CrossEntropyLoss(ignore_index=tag2idx["<pad>"])

for epoch in range(5):
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        logits = model(X_batch)                     # (batch, seq_len, num_tags)
        loss = loss_fn(logits.view(-1, len(tag_set)), y_batch.view(-1))
        loss.backward()
        optimizer.step()

Step 5: Evaluating — The Accuracy Paradox, Again, in a New Form

from sklearn.metrics import classification_report

print(classification_report(all_true_tags, all_predicted_tags))

Step 6: Extracting Structured Output From New Documents

def extract_entities(text, model, vocab, idx2tag):
    tokens = text.split()
    ids = torch.tensor([[vocab.get(t, 1) for t in tokens]])
    predicted_tags = model(ids).argmax(dim=-1)[0]

    entities = []
    current = []
    for token, tag_idx in zip(tokens, predicted_tags):
        tag = idx2tag[tag_idx.item()]
        if tag.startswith("B-"):
            if current:
                entities.append(" ".join(current))
            current = [token]
        elif tag.startswith("I-") and current:
            current.append(token)
        else:
            if current:
                entities.append(" ".join(current))
            current = []
    return entities

This is the genuine payoff of the B-/I- scheme from Step 1 — reassembling consecutive B-/I- tokens back into whole multi-word entity strings, turning a stream of per-token tags into the structured, readable output the chapter's own title promised.

What's Next

Chapter 7: Using an LLM API as a Data Labeling Tool — a light, deliberate touch of llm1/claude-adv1, using an LLM as one stage inside a larger data pipeline rather than training a model from scratch.

What We're Building

A pipeline that labels a large batch of unstructured customer-feedback text with a custom business category ("Billing Issue," "Bug Report," "Feature Request," "Praise," "Other") — categories with no existing labeled training data anywhere, making dsproj2-1's own from-scratch approach genuinely impractical here. The LLM applies the labeling scheme directly, at scale, the moment it can be described in a prompt — then the results get analyzed statistically with this track's own established tools.

Step 1: The Unlabeled Batch

import pandas as pd

feedback = pd.read_csv("customer_feedback.csv")   # one column: raw text, no labels at all
print(feedback.shape)

Step 2: Designing the Labeling Prompt

CATEGORIES = ["Billing Issue", "Bug Report", "Feature Request", "Praise", "Other"]

PROMPT_TEMPLATE = """Classify the following customer feedback into exactly one of these categories: {categories}.

Feedback: "{text}"

Respond with only valid JSON: {{"category": "...", "confidence": "high"|"medium"|"low"}}"""

Step 3: Calling the API — claude-adv1's Own Territory, Applied Minimally

import anthropic, json, time

client = anthropic.Anthropic()   # API key read from environment, per claude-adv1

def label_feedback(text):
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=100,
        messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(categories=CATEGORIES, text=text)}],
    )
    time.sleep(0.5)
    return json.loads(response.content[0].text)

The full API surface — streaming, tool use, system prompts — is claude-adv1's own territory in depth. This chapter uses exactly one call shape, deliberately, since the point here is what the API call accomplishes inside a larger pipeline, not the API itself.

Step 4: Running the Batch and Parsing Results

results = []
for text in feedback["text"]:
    try:
        label = label_feedback(text)
        results.append({"text": text, **label})
    except (json.JSONDecodeError, KeyError):
        results.append({"text": text, "category": "parse_error", "confidence": None})

labeled = pd.DataFrame(results)

Same JSON-parsing reflex ds1-3 established, now applied to a live model's own output instead of a file or a conventional API response — and the same defensive try/except instinct this track has used since dsproj1-1's own scraper, since a model occasionally returning malformed JSON is a real, expected possibility, not a bug to be surprised by.

Step 5: Analyzing the Labeled Data — This Track's Own Familiar Tools

import matplotlib.pyplot as plt

category_counts = labeled["category"].value_counts()
category_counts.plot(kind="barh", color="#15803D")
plt.title("Feedback Volume by Category")
plt.savefig("feedback_categories.png")

Nothing new here at all — .value_counts() and a bar chart, the exact ds1-level analysis skill this whole track opened with, applied here to data an LLM helped create rather than data that arrived pre-labeled.

The Honest Trade-Off, Stated Directly

What's Next

Chapter 8: Capstone — An End-to-End ML-Powered Mini App — collection, cleaning, feature engineering, model training/evaluation, and a minimal interface, combined into one real application, with a chapter-attribution table spanning both dsproj1 and dsproj2.

What We're Building

A "Churn Risk Checker" — reusing dsproj2-2's own churn-prediction pipeline in full (collect, clean, engineer features, train, evaluate), then wrapping the trained model in a minimal Streamlit interface: a small web form where a user enters a customer's own details and gets back a churn risk prediction, with no Python knowledge required to use it.

Stage 1 — Collect, Clean, Engineer (dsproj2-2, Reused)

import pandas as pd

df = pd.read_csv("telco_churn.csv")
df["avg_monthly_spend"] = df["TotalCharges"] / df["tenure"].replace(0, 1)
df = pd.get_dummies(df, columns=["Contract", "InternetService", "TechSupport"], drop_first=True)

y = (df["Churn"] == "Yes").astype(int)
X = df.drop(columns=["customerID", "Churn", "TotalCharges"])

Directly dsproj2-2's own Step 1, unchanged — nothing new to teach here, only to reuse.

Stage 2 — Train & Evaluate (dsproj2-2, Reused)

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))

Stage 3 — Persisting the Model (New)

import joblib

joblib.dump(model, "churn_model.joblib")
joblib.dump(list(X.columns), "model_columns.joblib")

Stage 4 — The Minimal Interface

# churn_app.py — run with: streamlit run churn_app.py
import streamlit as st
import pandas as pd
import joblib

model = joblib.load("churn_model.joblib")
columns = joblib.load("model_columns.joblib")

st.title("Churn Risk Checker")

tenure = st.slider("Tenure (months)", 0, 72, 12)
monthly_charges = st.slider("Monthly Charges ($)", 0, 150, 70)
contract = st.selectbox("Contract Type", ["Month-to-month", "One year", "Two year"])

if st.button("Check Risk"):
    row = pd.DataFrame([[0] * len(columns)], columns=columns)
    row["tenure"] = tenure
    row["MonthlyCharges"] = monthly_charges
    row["avg_monthly_spend"] = monthly_charges
    col_name = f"Contract_{contract}"
    if col_name in row.columns:
        row[col_name] = 1

    risk = model.predict_proba(row)[0][1]
    st.metric("Churn Risk", f"{risk:.0%}")

Streamlit turns each line into a real, interactive widget — st.slider(), st.selectbox(), st.button() — with no HTML, CSS, or JavaScript required. Building the input row with the exact saved columns order (Stage 3) is what lets a few manually-set values slot correctly into the same feature layout the model was trained on.

Chapter Attribution Table — Spanning Both Courses

Data Science & ML Subject Complete

That's all 16 chapters of the Data Science & ML Projects track — dsproj1's own 8 beginner projects and dsproj2's own 8 intermediate ones — and with it, the complete Data Science & ML subject: Data Science Fundamentals → Machine Learning Fundamentals → Neural Networks & Deep Learning → NLP → LLMs → Projects, six courses built one on top of the last, ending here with a real, working, interactive application built from every piece of it.