What We're Building

A scraper that: fetches a category listing page, follows its pagination across every page in that category (not just the first, unlike py4-7's own single-page version), visits each item's own detail page, downloads the linked content, and saves everything into a local folder plus a JSON manifest describing what was collected.

It's built and tested here against books.toscrape.com — the sister sandbox to py4-7's own quotes.toscrape.com, from the same project, explicitly built and maintained for scraping practice. The tool itself is generic by design: point the finished script at a real, permitted Linux-cheatsheet site (after checking that site's own robots.txt and Terms of Service, per this chapter's own warn-box) and it collects cheatsheets exactly the same way it collects practice-site book listings here.

Step 1: Fetching One Category's Listing Page

import requests
from bs4 import BeautifulSoup

CATEGORY_URL = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"

response = requests.get(CATEGORY_URL)
soup = BeautifulSoup(response.text, "html.parser")
items = soup.find_all("article", class_="product_pod")
print(len(items))   # one 
 per item on this page

Same requests.get() + BeautifulSoup(...).find_all() pattern py4-7 already introduced — one page fetched, one set of matching elements found.

Step 2: Following Pagination Across the Whole Category

import time

def get_all_items(start_url):
    items = []
    url = start_url
    while url:
        response = requests.get(url)
        soup = BeautifulSoup(response.text, "html.parser")
        items.extend(soup.find_all("article", class_="product_pod"))

        next_link = soup.find("li", class_="next")
        url = requests.compat.urljoin(url, next_link.find("a")["href"]) if next_link else None
        time.sleep(1)   # one polite pause per page, per this chapter's own warn-box

    return items

A while loop keeps following the page's own "Next →" link (soup.find("li", class_="next")) until there isn't one — None ends the loop. requests.compat.urljoin() turns the relative link on the page into a full URL. time.sleep(1) between requests is the same discipline py4-7's own warn-box named, applied here for real since this chapter actually fetches multiple pages.

Step 3: Visiting Each Item's Own Detail Page

def get_detail(item, base_url):
    link = item.find("h3").find("a")["href"]
    detail_url = requests.compat.urljoin(base_url, link)

    response = requests.get(detail_url)
    time.sleep(1)
    soup = BeautifulSoup(response.text, "html.parser")

    return {
        "title": soup.find("h1").get_text(),
        "price": soup.find("p", class_="price_color").get_text(),
        "url": detail_url,
    }

Each item on the listing page only has a title and a link — the fuller record lives on its own detail page. get_detail() follows that link and pulls out the specific fields worth keeping, exactly the "visit the linked page for the real content" step a real cheatsheet-collector would need.

Step 4: Saving the Manifest

import json
from pathlib import Path

def save_manifest(records, path="library_manifest.json"):
    Path(path).write_text(json.dumps(records, indent=2))
    print(f"Saved {len(records)} records to {path}.")

The same pathlib + json save pattern py4-7 used — a manifest is just data, and it persists the same way any other collected data does.

The Complete Scraper

import json, time, requests
from bs4 import BeautifulSoup
from pathlib import Path

START_URL = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"

def get_all_items(start_url):
    items, url = [], start_url
    while url:
        soup = BeautifulSoup(requests.get(url).text, "html.parser")
        items.extend(soup.find_all("article", class_="product_pod"))
        next_link = soup.find("li", class_="next")
        url = requests.compat.urljoin(url, next_link.find("a")["href"]) if next_link else None
        time.sleep(1)
    return items

def get_detail(item, base_url):
    link = item.find("h3").find("a")["href"]
    detail_url = requests.compat.urljoin(base_url, link)
    soup = BeautifulSoup(requests.get(detail_url).text, "html.parser")
    time.sleep(1)
    return {
        "title": soup.find("h1").get_text(),
        "price": soup.find("p", class_="price_color").get_text(),
        "url": detail_url,
    }

items = get_all_items(START_URL)
records = [get_detail(item, START_URL) for item in items]
Path("library_manifest.json").write_text(json.dumps(records, indent=2))
print(f"Saved {len(records)} records.")

What's Next

Chapter 2: Data Cleaning Pipeline: Wrangling a Messy Real-World CSV — turning ds1-4's own cleaning toolkit into a real, reusable pipeline, run against a genuinely messy dataset instead of a small illustrative one.

What We're Building

A clean_pipeline(df) function that takes a raw, messy customer-orders CSV, returns a cleaned DataFrame, and generates a human-readable cleaning report — a running log of every fix it made and why.

# customer_orders.csv — real-world messy, on purpose
order_id,customer_name,email,order_date,amount,country
1001," john smith ",john@example.com,2026-01-04,"$45.00",USA
1002,JANE DOE,jane@example.com,01/05/2026,"$120",uk
1003,Bob Lee,bob@example.com,2026-01-06,,USA
1001," john smith ",john@example.com,2026-01-04,"$45.00",USA
1004,Amy Chen,amy@example.com,2026-01-07,"$999999.99",
1005,dan miller,,2026-01-08,"eror",USA

Step 1: Load & Inspect First

import pandas as pd

df = pd.read_csv("customer_orders.csv")
print(df.info())
print(df.isna().sum())
print(df.duplicated().sum())

Same diagnostic reflex ds1-3/ds1-4 already established — .info() for dtypes, .isna().sum() for missing values per column, .duplicated().sum() for exact duplicate rows. Never clean before looking; the shape of the mess determines which fixes actually apply.

Step 2: Duplicates

def remove_duplicates(df, report):
    before = len(df)
    df = df.drop_duplicates()
    removed = before - len(df)
    report.append(f"Removed {removed} exact duplicate row(s).")
    return df

Every cleaning function here follows the same shape: do the fix, measure what changed, append a plain-English line to a shared report list. Row 1001 above is an exact duplicate — same order, same everything — the safest kind of row to simply drop.

Step 3: Fix Types & Formats

def fix_amount(df, report):
    before_missing = df["amount"].isna().sum()
    df["amount"] = (
        df["amount"].astype(str)
        .str.replace("$", "", regex=False)
    )
    df["amount"] = pd.to_numeric(df["amount"], errors="coerce")   # "eror" -> NaN
    coerced = df["amount"].isna().sum() - before_missing
    report.append(f"Converted amount to numeric; {coerced} unparseable value(s) became missing.")
    return df

def fix_dates(df, report):
    df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")
    report.append("Standardized order_date to a single datetime format.")
    return df

errors="coerce" is the load-bearing choice in both functions — instead of crashing on "eror" or an inconsistent date format, pandas turns anything it can't parse into NaN, which Step 5 then handles deliberately rather than letting a bad value silently corrupt a calculation.

Step 4: Standardize Text

def standardize_names(df, report):
    df["customer_name"] = df["customer_name"].str.strip().str.title()
    df["country"] = df["country"].str.strip().str.upper()
    report.append("Trimmed whitespace and standardized casing on customer_name and country.")
    return df

" john smith " and "JANE DOE" are two customers written two very different ways — .str.strip() removes stray whitespace, .str.title() gives every name the same casing convention, so two records for the same person don't silently look like two different customers to anything downstream.

Step 5: Missing Values — A Real Decision, Per Column

def handle_missing(df, report):
    df = df.dropna(subset=["customer_name"])   # can't analyze an order with no customer
    report.append("Dropped rows with a missing customer_name (unrecoverable identifier).")

    median_amount = df["amount"].median()
    n_filled = df["amount"].isna().sum()
    df["amount"] = df["amount"].fillna(median_amount)
    report.append(f"Filled {n_filled} missing amount value(s) with the median (${median_amount:.2f}).")

    n_unknown = df["country"].isna().sum()
    df["country"] = df["country"].fillna("UNKNOWN")
    report.append(f"Filled {n_unknown} missing country value(s) with 'UNKNOWN'.")
    return df

Step 6: Flagging (Not Silently Removing) Outliers

def flag_outliers(df, report):
    q1, q3 = df["amount"].quantile([0.25, 0.75])
    iqr = q3 - q1
    upper_bound = q3 + 1.5 * iqr   # the same IQR rule ds1-6 covers in full
    flagged = df[df["amount"] > upper_bound]
    report.append(f"Flagged {len(flagged)} row(s) as statistical outliers (amount > ${upper_bound:.2f}) for manual review — not removed.")
    return df, flagged

$999999.99 is almost certainly a data-entry error, not a genuine order — but this pipeline flags it for a human to check rather than deleting it automatically. ds1-6 covers the IQR rule's own full statistical reasoning; here it's applied as a practical, reusable check.

The Complete Pipeline

def clean_pipeline(input_path):
    df = pd.read_csv(input_path)
    report = [f"Started with {len(df)} rows."]

    df = remove_duplicates(df, report)
    df = fix_amount(df, report)
    df = fix_dates(df, report)
    df = standardize_names(df, report)
    df = handle_missing(df, report)
    df, flagged = flag_outliers(df, report)

    report.append(f"Finished with {len(df)} rows.")
    return df, flagged, report

cleaned, flagged, report = clean_pipeline("customer_orders.csv")
cleaned.to_csv("customer_orders_cleaned.csv", index=False)
print("\n".join(report))

What's Next

Chapter 3: Weather Data Dashboard — the first project pulling live data from an API instead of a static file, applying ds1-3's own JSON-handling material to real-time responses.

What We're Building

A small tool that fetches an hourly weather forecast from Open-Meteo — a genuinely free, public weather API that needs no signup or API key, which is exactly why it's used here — turns the response into a DataFrame, appends it to a growing local dataset, and builds two small charts from it: a temperature-over-time line chart and a precipitation-by-hour bar chart.

Step 1: Making the API Request

import requests

params = {
    "latitude": 51.51,     # London, as an example — swap for any location
    "longitude": -0.13,
    "hourly": "temperature_2m,precipitation",
    "forecast_days": 2,
}
response = requests.get("https://api.open-meteo.com/v1/forecast", params=params)
print(response.status_code)   # 200 — success

Passing a params dict to requests.get() builds the full query-string URL automatically — no manual string concatenation needed. No API key here, unlike most weather services, which is exactly why this API was chosen for a beginner project: nothing to configure, sign up for, or accidentally commit to a public repository.

Step 2: Parsing JSON From a Live Response

data = response.json()
print(list(data["hourly"].keys()))
# ['time', 'temperature_2m', 'precipitation']

Step 3: A Genuinely Different JSON Shape

import pandas as pd

hourly = data["hourly"]
df = pd.DataFrame({
    "time": pd.to_datetime(hourly["time"]),
    "temperature": hourly["temperature_2m"],
    "precipitation": hourly["precipitation"],
})
print(df.head())

Step 4: Storing Results Locally Over Time

from pathlib import Path

def append_to_log(df, path="weather_log.csv"):
    if Path(path).exists():
        df.to_csv(path, mode="a", header=False, index=False)
    else:
        df.to_csv(path, index=False)
    print(f"Appended {len(df)} rows to {path}.")

Running this script once a day builds a genuinely growing local weather history out of nothing but repeated forecast pulls — mode="a" appends rather than overwriting, and skipping the header on every call after the first keeps the CSV valid.

Step 5: Building the Dashboard

import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 6))

ax1.plot(df["time"], df["temperature"], color="#15803D")
ax1.set_title("Temperature Forecast")
ax1.set_ylabel("°C")

ax2.bar(df["time"], df["precipitation"], color="#EAB308")
ax2.set_title("Precipitation Forecast")
ax2.set_ylabel("mm")

plt.tight_layout()
plt.savefig("weather_dashboard.png")

ds1-7's own line-chart-for-a-trend and bar-chart-for-discrete-values choices, applied directly — temperature is a continuous value changing over time (a line), precipitation at each hour is a discrete quantity (a bar). Two subplots stacked in one figure make a small, genuine dashboard rather than two disconnected charts.

The Complete Script

import requests, pandas as pd, matplotlib.pyplot as plt
from pathlib import Path

params = {"latitude": 51.51, "longitude": -0.13,
          "hourly": "temperature_2m,precipitation", "forecast_days": 2}
data = requests.get("https://api.open-meteo.com/v1/forecast", params=params).json()

hourly = data["hourly"]
df = pd.DataFrame({
    "time": pd.to_datetime(hourly["time"]),
    "temperature": hourly["temperature_2m"],
    "precipitation": hourly["precipitation"],
})

log_path = "weather_log.csv"
df.to_csv(log_path, mode="a" if Path(log_path).exists() else "w",
          header=not Path(log_path).exists(), index=False)

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 6))
ax1.plot(df["time"], df["temperature"], color="#15803D")
ax1.set_title("Temperature Forecast")
ax2.bar(df["time"], df["precipitation"], color="#EAB308")
ax2.set_title("Precipitation Forecast")
plt.tight_layout()
plt.savefig("weather_dashboard.png")
print("Dashboard updated.")

What's Next

Chapter 4: Personal Finance Analyzer, Revisited With pandas — taking py4-8's own plain-Python expense tracker and rebuilding its analysis with the pandas/numpy toolkit this course has been using all along.

What We're Building

The same expense data py4-8 tracked — amount, category, description — now with one addition a real finance tool needs: a date per expense. Loaded into a DataFrame, the exact same two summaries py4-8 built by hand become one-line pandas operations, and two genuinely new analyses (monthly trends, a real chart) become possible that weren't practical to hand-roll before.

# expenses.json — py4-8's own schema, plus one new field: date
[
  {"date": "2026-01-03", "amount": 12.50, "category": "Food", "description": "Lunch"},
  {"date": "2026-01-05", "amount": 45.00, "category": "Transport", "description": "Fuel"},
  {"date": "2026-02-01", "amount": 89.99, "category": "Food", "description": "Groceries"}
]

Step 1: Loading Into a DataFrame

import pandas as pd

df = pd.read_json("expenses.json")
df["date"] = pd.to_datetime(df["date"])
print(df.head())

One line replaces py4-8's own json.loads(FILE.read_text()) plus manually keeping the result as a list of dicts — pd.read_json() parses the file directly into a DataFrame.

Step 2: Totals — The Direct Payoff

totals = {}
for e in expenses:
    totals[e["category"]] = (
        totals.get(e["category"], 0)
        + e["amount"]
    )

totals = df.groupby("category")["amount"].sum()

sum(e["amount"] for e in expenses)

df["amount"].sum()

py4-8's own totals.get(category, 0) + amount pattern was itself a direct reuse of the site's own word-frequency-counter idiom — real, working code. .groupby("category")["amount"].sum() does the identical job in one call, built on the same vectorized, C-level operations ds1-2 introduced as NumPy's own real advantage over a plain Python loop.

Step 3: What py4-8 Genuinely Couldn't Do Easily — Monthly Trends

monthly = df.set_index("date").resample("ME")["amount"].sum()
print(monthly)
# 2026-01-31    57.50
# 2026-02-28    89.99

Reproducing this in plain Python would mean manually parsing every date string, bucketing entries by year-and-month into a dict, and keeping that bucketing logic in sync everywhere it's needed — not impossible, but real, fiddly work py4-8's own scope never asked for. .resample("ME") does the entire "group by calendar month" operation in one call, because the DataFrame's own datetime index already understands calendar structure.

Step 4: A Real Chart

import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4.5))

totals.plot(kind="pie", ax=ax1, autopct="%1.0f%%", ylabel="",
            colors=["#15803D", "#EAB308", "#4ade80", "#facc15"])
ax1.set_title("Spending by Category")

monthly.plot(kind="line", marker="o", ax=ax2, color="#15803D")
ax2.set_title("Spending Over Time")
ax2.set_ylabel("$")

plt.tight_layout()
plt.savefig("finance_dashboard.png")

Both charts plot directly off the groupby/resample results from Steps 2 and 3 — a genuine payoff of keeping the data in a DataFrame throughout, rather than converting back to plain lists and dicts just to hand them to a chart.

The Complete Analysis Script

import pandas as pd, matplotlib.pyplot as plt

df = pd.read_json("expenses.json")
df["date"] = pd.to_datetime(df["date"])

totals = df.groupby("category")["amount"].sum()
monthly = df.set_index("date").resample("ME")["amount"].sum()

print("Totals by category:\n", totals)
print("\nMonthly spending:\n", monthly)
print(f"\nOverall total: ${df['amount'].sum():.2f}")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4.5))
totals.plot(kind="pie", ax=ax1, autopct="%1.0f%%", ylabel="")
ax1.set_title("Spending by Category")
monthly.plot(kind="line", marker="o", ax=ax2)
ax2.set_title("Spending Over Time")
plt.tight_layout()
plt.savefig("finance_dashboard.png")

What's Next

Chapter 5: Movie Ratings Explorer — the course's own central EDA chapter, applying ds1-9's/ds1-10's own seven-step methodology to a fresh public dataset without a checklist to lean on.

What We're Building

An exploration of a MovieLens-style ratings dataset — a real, well-known public research dataset of user ratings for movies, freely available and widely used for exactly this kind of practice. Two files: movies.csv (movie ID, title, genre, release year) and ratings.csv (user ID, movie ID, rating, timestamp). The question driving the whole exploration: what actually makes a movie "highly rated," and does the obvious answer hold up once the data gets a real look?

Loading and Merging

import pandas as pd

movies = pd.read_csv("movies.csv")
ratings = pd.read_csv("ratings.csv")

df = ratings.merge(movies, on="movie_id")
print(df.shape)
print(df.head())
print(df["rating"].describe())

The merge itself is ds1-5's own join material put to real use — a ratings file alone has no genre or title, and a movies file alone has no ratings; the interesting questions only exist once both are joined. .shape and .describe() here aren't a separate "Step 1" — they're just the first thing worth checking before trusting anything built on top of this data.

What Does a Rating Actually Look Like?

import seaborn as sns
import matplotlib.pyplot as plt

sns.histplot(df["rating"], bins=10, color="#15803D")
plt.title("Distribution of Individual Ratings")
plt.savefig("rating_distribution.png")

Ratings cluster heavily around 3-4 out of 5, with genuinely low ratings comparatively rare — a real, common pattern in rating data worth noticing before trusting any average built from it: the scale isn't used symmetrically, so "average rating" already means something skewed toward the positive end before any other analysis happens.

Which Genres Rate Highest?

genre_avg = df.groupby("genre")["rating"].mean().sort_values(ascending=False)
print(genre_avg)

genre_avg.plot(kind="barh", color="#EAB308")
plt.title("Average Rating by Genre")
plt.savefig("genre_ratings.png")

A one-line groupby, but already worth a second look before drawing conclusions — some genres will naturally have far fewer total ratings than others, which is exactly the kind of thing that turns into a genuine trap two sections from now.

Popular vs. Highly Rated — Not the Same Question

title_stats = df.groupby("title").agg(
    avg_rating=("rating", "mean"),
    num_ratings=("rating", "count"),
)

print("Top 10 by average rating:")
print(title_stats.sort_values("avg_rating", ascending=False).head(10))

print("\nTop 10 by number of ratings:")
print(title_stats.sort_values("num_ratings", ascending=False).head(10))

A Statistical Trap: Small-Sample Extremes

Why does a one-rating movie so easily top the list? With only one or two data points, the average has nothing to pull it toward a typical value — a single 5 or a single 1 is the entire average. As the number of ratings grows, the law of large numbers (ds1-6) pulls the average toward the movie's own genuine, underlying quality, since the extremes of individual opinion start to cancel out. This is the same small-sample-size caution ds1-6 raised in the abstract, now caught red-handed in a real dataset.

reliable = title_stats[title_stats["num_ratings"] >= 20]
print(reliable.sort_values("avg_rating", ascending=False).head(10))

Forming a Real Hypothesis: Do Older Movies Rate Higher?

df["decade"] = (df["year"] // 10) * 10
decade_avg = df.groupby("decade")["rating"].mean()

decade_avg.plot(kind="line", marker="o", color="#15803D")
plt.title("Average Rating by Decade")
plt.savefig("decade_trend.png")

What's Next

Chapter 6: Multi-Source Data Aggregator — combining two independent public APIs into one clean, merged dataset, extending Chapter 3's own single-API pattern to genuine multi-source integration.

What We're Building

A dataset combining country information from the REST Countries API (population, capital, currency — free, no signup) with live currency exchange rates from an open exchange-rate API (also free, no signup) — producing one table showing each country's population alongside its own currency's current value against the US dollar. Neither API alone can answer "which populous countries currently have the weakest currency" — only the combination can.

Step 1: Fetching From the First API

import requests
import pandas as pd

response = requests.get(
    "https://restcountries.com/v3.1/region/europe",
    params={"fields": "name,capital,population,currencies"},
)
countries_data = response.json()
print(len(countries_data), "countries returned")

Same requests.get(url, params={...}) pattern Chapter 3 introduced — just a different service, restricted to one region to keep this example a manageable size.

Step 2: Extracting a Currency Code From Nested JSON

records = []
for c in countries_data:
    currencies = c.get("currencies", {})
    currency_code = list(currencies.keys())[0] if currencies else None
    records.append({
        "country": c["name"]["common"],
        "capital": c.get("capital", [None])[0],
        "population": c["population"],
        "currency_code": currency_code,
    })

countries_df = pd.DataFrame(records)
print(countries_df.head())

Each country's currency arrives as a nested dict keyed by its own three-letter code (e.g. {"EUR": {...}}) — currency_code here isn't a field this API hands over directly, it's derived by pulling the first key out of that nested structure. This derived value is exactly what makes the merge in Step 4 possible at all.

Step 3: Fetching From the Second, Unrelated API

rates_response = requests.get("https://open.er-api.com/v6/latest/USD")
rates_data = rates_response.json()["rates"]

rates_df = pd.DataFrame(list(rates_data.items()), columns=["currency_code", "usd_rate"])
print(rates_df.head())

This API has never heard of REST Countries and doesn't organize its data by country at all — it's a flat dict of currency codes to exchange rates. rates_data.items() turns that dict into a list of (code, rate) pairs, which becomes a two-column DataFrame — the same currency-code column countries_df already has, arrived at completely independently.

Step 4: The Merge — ds1-5's Own Material, on a Derived Key

combined = countries_df.merge(rates_df, on="currency_code", how="left")
print(combined.sort_values("population", ascending=False).head(10))

A Quick Combined Insight

top_by_population = combined.sort_values("population", ascending=False).head(10)
print(top_by_population[["country", "population", "currency_code", "usd_rate"]])

A table like this — population from one API, live currency strength from a completely different one — genuinely doesn't exist as a single downloadable file anywhere. It only exists because this chapter built it, on the spot, from two independent sources.

The Complete Aggregator

import requests, pandas as pd

countries_data = requests.get(
    "https://restcountries.com/v3.1/region/europe",
    params={"fields": "name,capital,population,currencies"},
).json()

records = []
for c in countries_data:
    currencies = c.get("currencies", {})
    records.append({
        "country": c["name"]["common"],
        "capital": c.get("capital", [None])[0],
        "population": c["population"],
        "currency_code": list(currencies.keys())[0] if currencies else None,
    })
countries_df = pd.DataFrame(records)

rates_data = requests.get("https://open.er-api.com/v6/latest/USD").json()["rates"]
rates_df = pd.DataFrame(list(rates_data.items()), columns=["currency_code", "usd_rate"])

combined = countries_df.merge(rates_df, on="currency_code", how="left")
combined.to_csv("country_currency_combined.csv", index=False)
print(f"Combined {len(combined)} countries across two independent APIs.")

What's Next

Chapter 7: A First Real Classifier — this course's own single, deliberately light touch of ml1, a small, approachable classification project on a well-known dataset.

What We're Building

A classifier for the Iris dataset — the single most famous "hello world" of classification: predict which of three flower species a sample belongs to, from four simple measurements (sepal length, sepal width, petal length, petal width). It ships built directly into scikit-learn, needs zero cleaning, and is small enough to see the whole pipeline at once.

Step 1: Loading the Dataset

from sklearn.datasets import load_iris
import pandas as pd

iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df["species"] = [iris.target_names[i] for i in iris.target]
print(df.head())

iris.data is a NumPy array of the four measurements; iris.target is the numeric species label for each row (0, 1, or 2). Building a DataFrame and mapping the numbers back to real species names makes the data readable the same way every prior chapter's own data has been.

Step 2: A Quick Look — Why This Dataset Works So Well

import seaborn as sns
import matplotlib.pyplot as plt

sns.scatterplot(
    data=df, x="petal length (cm)", y="petal width (cm)",
    hue="species", palette=["#15803D", "#EAB308", "#4ade80"],
)
plt.title("Petal Measurements by Species")
plt.savefig("iris_scatter.png")

Plotting just two of the four features already shows the three species separating into clearly distinct clusters — a genuine, visible reason this particular dataset became the classic teaching example: the classes are almost perfectly separable, letting even a simple model succeed clearly enough to see what's actually happening.

Step 3: Train/Test Split

from sklearn.model_selection import train_test_split

X = df[iris.feature_names]
y = df["species"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(f"Training on {len(X_train)} rows, testing on {len(X_test)} rows.")

The full reasoning behind why a model needs to be evaluated on data it never saw during training is ml1-2's own material in full — used correctly here, not re-derived. random_state=42 just makes the split reproducible from one run to the next.

Step 4: Fitting a Simple Classifier

from sklearn.neighbors import KNeighborsClassifier

model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)

Step 5: Evaluating — The Single Simplest Metric

accuracy = model.score(X_test, y_test)
print(f"Accuracy: {accuracy:.2%}")

Step 6: Seeing the Predictions

predictions = model.predict(X_test)
results = X_test.copy()
results["actual"] = y_test.values
results["predicted"] = predictions
results["correct"] = results["actual"] == results["predicted"]

print(results[results["correct"] == False])   # the rows the model got wrong, if any

Looking directly at whichever rows the model got wrong (if any) is worth more than the single accuracy number alone — on this dataset, mistakes almost always happen between two of the three species that sit closest together in the Step 2 scatter plot, not randomly across all three, a small, concrete confirmation that the model is making genuinely sensible errors rather than arbitrary ones.

The Complete Classifier

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
import pandas as pd

iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df["species"] = [iris.target_names[i] for i in iris.target]

X = df[iris.feature_names]
y = df["species"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)

accuracy = model.score(X_test, y_test)
print(f"Accuracy: {accuracy:.2%}")

What's Next

Chapter 8: Capstone — Your Own Dataset, Start to Finish — no fixed dataset this time: your own choice, taken through the complete collect-clean-explore-visualize-model pipeline this course has built one piece at a time.

The Pipeline, and What It's Demonstrated On

Palmer Penguins — real physical measurements (bill length, bill depth, flipper length, body mass) of three penguin species from Palmer Station, Antarctica, plus island and sex. It's used here specifically because, unlike Chapter 7's own squeaky-clean Iris dataset, it has real missing values and a real categorical column — giving this capstone a genuine reason to exercise every stage of the pipeline, not just the modeling step.

Stage 1 — Collect

import pandas as pd

# swap this line for your own dataset:
#   - a downloaded CSV (as here)
#   - a scraped page (Chapter 1's own pattern)
#   - a live API, single or combined (Chapter 3's / Chapter 6's own pattern)
raw = pd.read_csv("penguins.csv")
print(raw.shape)

Stage 2 — Clean

def clean_penguins(df, report):
    before = len(df)
    df = df.dropna(subset=["species", "bill_length_mm"])
    report.append(f"Dropped {before - len(df)} row(s) missing species or bill_length_mm.")

    n_sex_missing = df["sex"].isna().sum()
    df["sex"] = df["sex"].fillna("unknown")
    report.append(f"Filled {n_sex_missing} missing sex value(s) with 'unknown'.")

    return df

report = [f"Started with {len(raw)} rows."]
cleaned = clean_penguins(raw, report)
report.append(f"Finished cleaning with {len(cleaned)} rows.")

Directly reusing Chapter 2's own shape: a function that cleans and appends plain-English lines to a shared report list, with the same kind of real per-column judgment call that chapter's own warn-box named — rows missing a core measurement are dropped, a missing category is labeled rather than guessed.

Stage 3 — Explore

report.append("--- Exploration ---")
report.append(str(cleaned.groupby("species")["bill_length_mm"].describe()[["mean", "std"]]))
report.append(f"Species counts: {cleaned['species'].value_counts().to_dict()}")

Chapter 5's own flowing, narrative style rather than a numbered checklist — a couple of well-chosen summaries, not an exhaustive re-run of every EDA step, since the real goal here is a working pipeline end to end, not a second full exploration chapter.

Stage 4 — Visualize

import seaborn as sns
import matplotlib.pyplot as plt

sns.scatterplot(
    data=cleaned, x="bill_length_mm", y="flipper_length_mm",
    hue="species", palette=["#15803D", "#EAB308", "#4ade80"],
)
plt.title("Bill Length vs. Flipper Length by Species")
plt.savefig("capstone_dashboard.png")

Stage 5 — Model

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

features = ["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"]
X = cleaned[features].dropna()
y = cleaned.loc[X.index, "species"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)
accuracy = model.score(X_test, y_test)

report.append(f"--- Model ---")
report.append(f"KNN classifier accuracy on held-out test data: {accuracy:.2%}")

Chapter 7's own exact template — same algorithm, same honest framing (a light touch, not a substitute for ml1's own full depth) — applied here to a dataset with genuinely more real-world texture than Iris had.

Stage 6 — Report

from pathlib import Path

Path("findings_report.txt").write_text("\n".join(report))
print("\n".join(report))

The same report-list pattern from Chapter 2, now spanning the entire pipeline rather than just the cleaning stage — one readable, saved artifact documenting exactly what happened to the data at every step, from raw file to trained model.

Chapter Attribution Table

Course Complete

That's all 8 projects of Data Science & ML Projects (Beginner) — a scraper, a cleaning pipeline, a live weather dashboard, a pandas-powered finance analyzer, a full EDA, a multi-source API aggregator, a first classifier, and this capstone — combining ds1's own data-handling toolkit with one light, honest taste of ml1. The still-unfleshed dsproj2 (Intermediate) remains as the final piece of the Data Science & ML subject's own six-course roadmap, and is exactly where NLP-shaped projects like a real sentiment analyzer belong.