This course assumes real, working Python fluency — py1–py4 already cover variables, control flow, functions, files, JSON, and packaging in depth, and none of that gets re-taught here. This chapter picks up almost exactly where py1-9's own brief pip/virtual-environment coverage left off, moving from "how to manage Python packages in general" into the specific stack this course is actually about.

Data Science Is Not a Synonym for Machine Learning

Four terms get used almost interchangeably in casual conversation, and they shouldn't be: Business Intelligence (BI) is retrospective reporting on known questions — dashboards showing what already happened. Data Analytics is answering a specific business question with data — "did the redesign increase signups?" Machine Learning is building models that make predictions or find patterns automatically from data. Data Science is the broader discipline that combines statistics, programming, and domain knowledge to extract insight from data — and machine learning is one tool inside that discipline, not a synonym for the whole thing.

The Data Science Workflow

Five stages, in order, form the backbone this whole subject is organized around:

Jupyter Notebooks — The Standard Data Science Environment

py1-1 introduced running Python as a plain script, top to bottom, once. Data science work is exploratory and iterative by nature — you look at a dataset, try something, look at the result, adjust, try again — and a plain script re-run from the top every time that happens is genuinely slow to work with. A Jupyter notebook is a document made of individual, independently runnable cells: code cells that keep their own output (including inline charts) visible directly beneath them, and text cells for notes, mixed freely in one file. Running one cell at a time, keeping earlier results in memory, and only re-running what actually changed is the standard way real data science work gets done — not a replacement for py1's own script model in general, just the better-fitting tool for this specific, exploratory kind of work.

The Python Data Science Stack — A First Look

A Short, Grounding Example

A retail company wants to know why weekend sales dipped last month. A data scientist starts by collecting the relevant sales records (a database export, in this case — ds1-3's own CSV-reading skills apply directly). The raw export has missing store IDs and a few duplicated rows from a failed sync — cleaning (ds1-4) fixes that before anything else happens. Exploring the cleaned data (ds1-5–ds1-9) reveals the dip is concentrated at one specific store, not company-wide — a pattern no amount of staring at raw numbers would have surfaced as quickly as a grouped chart does. No model was ever built; the answer came entirely from disciplined cleaning and exploration. Communicating that finding — one clear chart, one clear sentence — closes the loop.

Hands-On Exercises

ds1-1's own stack roadmap named NumPy as the first library this course actually delivers in depth. Everything in the rest of this course — pandas' own DataFrame (ds1-3), the statistics chapter (ds1-6), even the plotting libraries (ds1-7, ds1-8) — is built directly on top of what this chapter introduces.

The ndarray vs. py1-6's Own Python List

py1-6 covered the Python list: flexible, resizable, and able to hold mixed types in the same container (a string, an integer, and another list, all in one list, is perfectly valid Python). NumPy's core data structure, the ndarray (n-dimensional array), makes the opposite trade-off deliberately: every element must be the same fixed type, and — while an array can be resized — its whole point is to represent a fixed-shape block of homogeneous numerical data efficiently.

import numpy as np

arr = np.array([1, 2, 3, 4, 5])
zeros = np.zeros(5)          # array([0., 0., 0., 0., 0.])
ones = np.ones((2, 3))       # a 2x3 array of 1.0
ranged = np.arange(0, 10, 2) # array([0, 2, 4, 6, 8])

Vectorization — Why This Trade-off Exists

Compare summing every element of a million-item collection, squared, using py1-6's own list-and-loop approach against NumPy's own vectorized equivalent:

total = 0
for x in my_list:            # plain Python loop
    total += x ** 2

total = (my_array ** 2).sum()  # NumPy — no explicit loop at all

The vectorized version isn't just shorter — it's genuinely, dramatically faster on large data, often by one to two orders of magnitude. The reason is structural, not a matter of Python's own loop syntax being slow in some vague sense: a Python list stores a collection of separate objects scattered in memory, and a plain Python for loop has to check each element's type and dispatch the right operation for it, one element at a time, through Python's own interpreter overhead. A NumPy array stores its elements as one single, contiguous, fixed-type block of memory, and a vectorized operation like ** 2 runs as one single, compiled, low-level loop over that block — no per-element type-checking, no per-element interpreter dispatch.

Broadcasting

Broadcasting lets NumPy apply an operation between two arrays of different, but compatible, shapes — without writing an explicit loop to line them up. The simplest case: adding a single number to an entire array applies that number to every element.

arr = np.array([1, 2, 3])
arr + 10        # array([11, 12, 13]) — the scalar is "broadcast" across every element

matrix = np.array([[1, 2, 3], [4, 5, 6]])
row = np.array([10, 20, 30])
matrix + row    # row is broadcast across each row of matrix

The compatibility rule, informally: comparing shapes from the trailing dimension backward, each pair of dimensions must either match exactly or one of them must be 1. A (2, 3) array and a (3,) array are compatible (the trailing dimensions both read 3); a (2, 3) array and a (2,) array are not, without reshaping first.

Indexing, Slicing & Boolean Masking

Basic indexing and slicing work much like py1-6's own list slicing (arr[1:4], arr[-1]). NumPy adds a capability plain lists don't have at all: boolean masking — indexing an array with a condition that evaluates to an array of True/False values, keeping only the elements where the condition holds.

arr = np.array([3, 7, 2, 9, 4])
arr > 5              # array([False, True, False, True, False])
arr[arr > 5]          # array([7, 9]) — only the matching elements

Basic Linear Algebra

A quick preview, not a full course: arr.reshape(2, 3) changes an array's shape without changing its data; arr.T transposes a 2D array; a @ b (or np.dot(a, b)) performs matrix multiplication. These three operations alone are the mathematical backbone of how a neural network actually computes a prediction — a fact nn1 will build on directly once this course is complete.

Hands-On Exercises

ds1-2 built the ndarray. This chapter delivers the second row of ds1-1's own stack roadmap: pandas, the library this entire course's own idea of "a dataset" is actually built around. Every remaining chapter through ds1-9 works with the two structures introduced here.

Series — A Labeled ndarray

A pandas Series is, structurally, exactly what ds1-2's own ndarray already was, plus one addition: a labeled index alongside the values.

import pandas as pd

s = pd.Series([120, 340, 95, 410], index=["Mon", "Tue", "Wed", "Thu"])
s["Tue"]      # 340 — access by label
s.values      # array([120, 340, 95, 410]) — the underlying ndarray, unchanged from ds1-2

That last line is worth pausing on: .values hands back a genuine NumPy ndarray. A Series isn't a new kind of data underneath — it's an ndarray with labels attached on top.

DataFrame — A Table of Aligned Series

A DataFrame extends the same idea to two dimensions: a table where each column is its own Series, and every column shares the same row index. ds1-1's own throughline promised this explicitly — a DataFrame is a labeled wrapper around an ndarray, not a separate structure invented from scratch.

df = pd.DataFrame({
    "product": ["Latte", "Espresso", "Latte", "Mocha"],
    "quantity": [12, 8, 15, 6],
    "revenue": [54.00, 24.00, 67.50, 30.00],
})
df.values     # a 2D ndarray underneath — same relationship as Series.values above

Reading Real Data — CSV and JSON

py2-7 covered Python's own json module — json.load() parsing a file into nested dicts and lists by hand, followed by manually walking that structure to pull out the fields you actually want. At dataset scale, pandas replaces that manual walk with a single call:

df = pd.read_csv("sales.csv")
df = pd.read_json("sales.json")   # the py2-7 json.load() step, plus the table-construction step, in one call

.loc vs. .iloc — A Genuinely Common Beginner Confusion

Both select rows and columns. The difference is what they select by:

Boolean Filtering — ds1-2's Own Mechanism, One Layer Up

df[df["quantity"] > 10]   # only rows where quantity exceeds 10

This is ds1-2's own boolean masking, applied to whole rows instead of individual array elements — df["quantity"] > 10 produces a Series of True/False values (one per row), and indexing the DataFrame with it keeps only the matching rows. Exactly the pattern ds1-2's own warn-box flagged as worth remembering.

A First Look, Column Selection & Derived Columns

df.head()      # first 5 rows — a quick look
df.info()      # column types, non-null counts
df.describe()  # summary statistics — previews ds1-6

df["revenue"]                 # select one column (a Series)
df["avg_price"] = df["revenue"] / df["quantity"]   # add a computed column

This Course's Own Running Dataset

Picking up ds1-1's own retail example directly: a small daily sales log for a chain of coffee shops. This exact table — messy details included — is what ds1-4 spends its own chapter cleaning up, so nothing here needs fixing yet.

Four real, distinct problems, deliberately left visible here rather than fixed on sight: a missing store_id, an inconsistent date format on row 1004, an inconsistent product-name capitalization on row 1005, and a fully duplicated row at the end. ds1-4 addresses each one by name.

Hands-On Exercises

ds1-3 introduced a small coffee-shop sales table with four real, named problems, left deliberately unfixed. This chapter fixes each one — the same dataset, not a fresh example — so the techniques below are demonstrated on a problem already familiar rather than an abstract new one.

Problem 1: Missing Data — Row 1003's store_id

df.isna()            # a same-shape DataFrame of True/False, marking every missing cell
df["store_id"].isna().sum()   # how many missing values in just this column

df.dropna(subset=["store_id"])          # option A: drop rows with a missing store_id
df["store_id"].fillna("UNKNOWN")        # option B: fill missing values with a placeholder

Problem 2: The Duplicated Row

df.duplicated()          # True for each row that's an exact repeat of an earlier one
df.drop_duplicates()     # keeps the first occurrence, drops the rest

Problem 3: The Inconsistent Date Format

df["date"] = pd.to_datetime(df["date"])

Problem 4: Inconsistent Capitalization

df["product"] = df["product"].str.title()   # "latte" -> "Latte"

A First Look at Outlier Detection

A quick, practical rule of thumb, not yet the full statistical grounding (ds1-6 covers that properly): a value is a plausible outlier if it falls more than 1.5 × the interquartile range (IQR) below the 25th percentile or above the 75th percentile.

q1, q3 = df["revenue"].quantile([0.25, 0.75])
iqr = q3 - q1
outliers = df[(df["revenue"] < q1 - 1.5*iqr) | (df["revenue"] > q3 + 1.5*iqr)]

This is a mechanical flag, not an automatic verdict — a flagged value might be a genuine data-entry error, or it might be a real, unusually large sale that's actually the most interesting row in the whole dataset. ds1-6's own statistics chapter explains exactly what a quartile and an IQR are, and why 1.5× specifically is the conventional threshold.

Before & After

Hands-On Exercises

ds1-4 left the coffee-shop sales table clean: a consistent store_id, real datetime values, uniform product names, no duplicates. This chapter combines and reshapes that same cleaned table — and for a reader who already knows SQL (mysql2, mysql3), most of what follows is genuinely familiar material wearing pandas' own syntax.

groupby — Pandas' Own GROUP BY

mysql_foundations_07 (mysql2's own Aggregate Functions and GROUP BY chapter) covered grouping rows and computing an aggregate per group in SQL. Pandas' groupby does the exact same conceptual job:

df.groupby("product")["revenue"].sum()
# SQL equivalent: SELECT product, SUM(revenue) FROM sales GROUP BY product;

df.groupby("product").agg(total_revenue=("revenue", "sum"), orders=("order_id", "count"))
# SQL equivalent: SELECT product, SUM(revenue) AS total_revenue, COUNT(order_id) AS orders
#                 FROM sales GROUP BY product;

.agg() runs multiple aggregations at once, each labeled — the same shape as writing several aggregate functions in one SQL SELECT list.

merge — Pandas' Own JOIN

Introducing a second, small table — a store lookup, mapping store_id to a human-readable name and city:

merged = pd.merge(df, stores_df, on="store_id", how="left")
# SQL equivalent: SELECT * FROM sales LEFT JOIN stores ON sales.store_id = stores.store_id;

That "UNKNOWN" store_id from ds1-4 — kept, not dropped, precisely because a "left" join was used — now has no match in stores_df, and its store_name/city columns come back as NaN rather than the row disappearing. This is exactly the missing-data situation ds1-4 already covered, arriving again from a genuinely different source.

concat — Stacking, Not Matching

merge combines two tables based on matching key values. concat does something structurally different: it glues tables together along an axis, with no matching logic at all — useful for a case like combining July's sales file with August's, where both files share identical columns and just need to be stacked into one longer table.

full_year = pd.concat([july_df, august_df])   # stacked vertically, row-wise

Pivot Tables — Reshaping Long to Wide

A reader who's built a pivot table in a spreadsheet already knows this concept — pandas' own version does the same reshaping, in code:

pd.pivot_table(merged, index="date", columns="product", values="revenue", aggfunc="sum")

This is genuinely just groupby plus a reshape, combined into one call: it groups by two things at once (date and product), aggregates revenue within each combination, and then lays the result out as a real table — one row per date, one column per product — rather than the single flat column of grouped results groupby alone produces.

Putting It Together

merged = pd.merge(df, stores_df, on="store_id", how="left")
summary = merged.groupby(["store_name", "product"]).agg(
    total_revenue=("revenue", "sum"),
    orders=("order_id", "count"),
)

One merge, one grouped aggregation — a real per-store, per-product summary table, the exact shape of analysis ds1-9's own EDA chapter will lean on directly to actually spot patterns in a dataset, rather than just describing the mechanics of getting there.

Hands-On Exercises

ds1-4 used quartiles and an IQR without fully explaining either, promising this chapter would. This is that chapter — and everything here is deliberately vocabulary-first: ds1-7 through ds1-9 all lean on the terms defined below without redefining them.

Mean, Median & Mode — And Why the Choice Matters

Mean is the familiar average — sum divided by count. Median is the middle value once everything is sorted. Mode is the most frequent value. They usually sit close together — until a dataset has extreme values, at which point they diverge in a way that actually matters.

Spread — Delivering on ds1-4's Own IQR Promise

Range is simply max minus min — the crudest measure of spread, and highly sensitive to a single extreme value. Variance is the average of each value's squared distance from the mean; standard deviation is variance's own square root, put back into the original units so it's directly interpretable.

Quartiles split sorted data into four equal-sized groups: Q1 (the 25th percentile) is the value below which a quarter of the data falls; Q2 is the median itself; Q3 (the 75th percentile) is the value below which three-quarters of the data falls. IQR (interquartile range) is simply Q3 − Q1 — the spread of the middle half of the data, deliberately ignoring the most extreme quarter on each end. ds1-4's own 1.5×IQR threshold is a long-standing statistical convention (originating with the box plot itself) for marking a value as unusually far outside that middle-half range — not an arbitrary number, but not a law of nature either, just a widely agreed default.

Distributions & the Normal Curve

A distribution describes the overall shape of how values are spread across their possible range — not any single number, but the whole pattern. The normal distribution (the familiar symmetric "bell curve") is the single most important reference shape in statistics, fully described by just two numbers: its mean (the center) and its standard deviation (how wide the bell is). The empirical rule gives a fast, useful approximation for normally distributed data: roughly 68% of values fall within one standard deviation of the mean, roughly 95% within two, and roughly 99.7% within three.

Correlation vs. Causation

Correlation measures how strongly two variables move together, typically as a number from −1 to 1 (Pearson's correlation coefficient): 1 means they move in perfect lockstep upward together, −1 means one rises exactly as the other falls, 0 means no linear relationship at all. ds1-7's own scatter plots are the standard way to actually see a correlation visually.

Probability — Just Enough, Not a Full Course

A probability is a number from 0 to 1 describing how likely an event is. Two events are independent if one happening tells you nothing about whether the other happens (two separate coin flips); they're dependent otherwise (drawing two cards from the same deck without replacement — the first draw changes what's left for the second). This is deliberately the full extent of probability covered here — enough vocabulary to understand later statistical statements, not a dedicated probability course.

Sampling

A population is every member of the group actually being studied; a sample is the subset actually measured, because measuring an entire population is usually impossible or impractical. A sample is only useful if it's representative — if it resembles the population in the ways that matter for the question being asked.

Hands-On Exercises

ds1-6 built a shared statistical vocabulary — mean, distribution, correlation — entirely in words and numbers. This chapter starts turning that vocabulary into pictures. Matplotlib is the foundational plotting library the entire Python data-visualization ecosystem, including ds1-8's own Seaborn, is built directly on top of.

The Figure/Axes Model

Matplotlib's core object hierarchy has two parts worth knowing by name: a Figure is the whole canvas — the entire window or image being produced. An Axes is one individual plot area within that figure, with its own x-axis, y-axis, and content. A figure can hold one Axes, or several arranged in a grid.

import matplotlib.pyplot as plt

fig, ax = plt.subplots()      # one Figure, one Axes
ax.plot(df["date"], df["revenue"])
ax.set_title("Daily Revenue")
ax.set_xlabel("Date")
ax.set_ylabel("Revenue ($)")
plt.show()

Four Chart Types, Four Jobs

Line Plot — Revenue Over Time

daily = df.groupby("date")["revenue"].sum()
fig, ax = plt.subplots()
ax.plot(daily.index, daily.values)
ax.set_title("Daily Revenue Over Time")

Bar Plot — Revenue by Product

by_product = df.groupby("product")["revenue"].sum()
fig, ax = plt.subplots()
ax.bar(by_product.index, by_product.values)
ax.set_title("Total Revenue by Product")

Scatter Plot — Quantity vs. Revenue

fig, ax = plt.subplots()
ax.scatter(df["quantity"], df["revenue"])
ax.set_xlabel("Quantity")
ax.set_ylabel("Revenue ($)")

A visibly upward-sloping cluster of points here would be the visual signature of a strong positive correlation — ds1-6's own Pearson coefficient close to 1. ds1-6's own confounding-variable warning still applies at full strength here: a clear pattern in a scatter plot is a reason to investigate, never proof of a direct causal link on its own.

Histogram — The Shape of Order Revenue

fig, ax = plt.subplots()
ax.hist(df["revenue"], bins=10)
ax.set_title("Distribution of Order Revenue")

This is the chart that would actually make ds1-6's own catering-order outlier visible at a glance — a tall cluster of ordinary-sized orders, with one isolated bar sitting far off to the right.

Choosing the Right Chart Type

Hands-On Exercises

ds1-7's own tip-box promised this chapter's material would "look like a picture of exactly the numbers defined here [ds1-6], not new material." Seaborn earns that promise directly — every function below returns an ordinary Matplotlib Axes object underneath (ds1-7's own foundation, unchanged), with statistical plot types and DataFrame-aware syntax layered on top.

What Seaborn Actually Adds

Three genuine additions over plain Matplotlib: statistical chart types (box, violin, heatmap, pair plots) that ds1-7's own line/bar/scatter/histogram set doesn't include directly; nicer default styling out of the box; and DataFrame-aware syntax — passing a DataFrame plus column names directly, rather than raw arrays pulled out by hand first.

import seaborn as sns

sns.boxplot(data=df, y="revenue")   # DataFrame + column name, not a raw array

Box Plots — ds1-6's Own Vocabulary, Drawn Directly

This is the direct payoff of ds1-6's own promise. Every visual element of a box plot maps onto a term already defined:

The catering order from ds1-4/ds1-6 would appear here as exactly that: an isolated dot, sitting alone past the upper whisker, visually confirming the same flag the mechanical 1.5×IQR calculation already raised.

Violin Plots — Adding Shape to the Same Summary

sns.violinplot(data=df, y="revenue")

A violin plot shows the same quartile information a box plot does, mirrored into a smoothed, rotated shape — effectively ds1-7's own histogram, reshaped and reflected, layered directly over the box plot's own summary statistics. This matters specifically when a distribution's shape carries real information a box plot alone hides: two distinct clusters of typical order sizes (a genuinely bimodal distribution — small individual orders and separate large group orders, with few orders in between) can produce a perfectly ordinary-looking box plot, since a box plot only ever shows five summary numbers, never the shape connecting them. A violin plot would show that same data as two visible bulges, immediately revealing structure the box plot's own five numbers couldn't.

Heatmaps — Every Pairwise Correlation, One Chart

ds1-7's own scatter plot showed one pair of variables at a time. A correlation heatmap extends that to every numeric column at once:

correlations = df[["quantity", "revenue"]].corr()
sns.heatmap(correlations, annot=True, cmap="coolwarm")

df.corr() computes ds1-6's own Pearson coefficient for every pair of numeric columns simultaneously, producing a square matrix. sns.heatmap() renders that matrix as color intensity — strong positive correlations in one color, strong negative in another, weak correlations pale — with annot=True printing the actual coefficient value inside each cell. With only two numeric columns this collapses to a single interesting cell; the real value appears once a dataset has several numeric columns worth comparing pairwise all at once, exactly the situation ds1-9's own EDA methodology will lean on this for.

Pair Plots — ds1-7's Own Toolkit, Automated

sns.pairplot(df[["quantity", "revenue"]])

sns.pairplot() generates a full grid: a scatter plot for every pair of numeric columns (ds1-7's own scatter plot, run automatically for every combination) plus a histogram for each column against itself along the diagonal (ds1-7's own histogram, likewise automated). It's genuinely everything ds1-7 taught individually, generated for an entire dataset's worth of column pairs in one function call rather than one deliberate chart at a time.

Hands-On Exercises

ds1-3 through ds1-8 built a toolkit — reading data, cleaning it, combining it, describing it, plotting it — one technique at a time, on one running coffee-shop dataset. This chapter formally names the method those tools were all building toward, and deliberately applies it to a fresh dataset the coffee-shop material never touched: a small used-car listings table. The method, not a memorized sequence of steps on one specific dataset, is what this chapter needs to transfer — which is exactly why ds1-10's own capstone will apply this same seven-step structure to a third, still-different dataset.

The Seven-Step Methodology

Steps 3–5, Worked

df["price"].plot(kind="hist")                          # univariate — ds1-7's histogram
df["fuel_type"].value_counts()                          # univariate — categorical breakdown

df.plot(kind="scatter", x="mileage", y="price")          # bivariate — ds1-7's scatter plot
sns.boxplot(data=df, x="fuel_type", y="price")           # bivariate — ds1-8's box plot, grouped

sns.heatmap(df[["year","mileage","price"]].corr(), annot=True)   # multivariate — ds1-8's heatmap
sns.pairplot(df[["year","mileage","price"]])                     # multivariate — ds1-8's pair plot

The mileage-vs-price scatter plot here shows something the coffee-shop dataset's own quantity-vs-revenue scatter (ds1-7) never did: a clear negative correlation — higher mileage, lower price. Same tool, same underlying Pearson mechanism (ds1-6), a genuinely different real-world relationship.

Step 6: Spotting Anomalies, Honestly

Running ds1-4's own IQR rule on price flags the 1968 Jaguar E-Type immediately — its price sits far above every other listing. Exactly the honest distinction ds1-4 and ds1-6 already insisted on: this isn't a data-entry error to delete. A genuine vintage collector's car legitimately commands a price with no relationship at all to the mileage/age/price pattern the rest of the dataset follows — flagging it is correct; discarding it would be a real mistake, and would also quietly remove the row most worth asking a follow-up question about.

Step 7: Forming Hypotheses — The Actual Point of EDA

Everything above describes what the data is. This step turns that description into specific, testable questions:

• "Does mileage predict price more strongly than the car's year does?" — a question the heatmap's own two separate correlation values already give a first, rough answer to.
• "Is the Jaguar a genuinely different category of listing (collectible) that should be modeled separately from ordinary used cars?"
• "Does fuel type meaningfully shift price once mileage and year are accounted for?"

Hands-On Exercises

A third dataset, unrelated to the coffee-shop sales table (ds1-3–ds1-8) or the used-car listings (ds1-9): a small employee-attrition table for a fictional company, asking a genuinely different kind of question — not "what predicts a price," but "what's associated with an employee leaving." ds1-9's own seven-step methodology, applied start to finish, unchanged.

Step 1 — Shape

Step 2 — Summary Statistics

Step 3 — Univariate Analysis

df["years_at_company"].plot(kind="hist")
df["department"].value_counts()
df["left_company"].value_counts()

Step 4 — Bivariate Analysis

df.plot(kind="scatter", x="years_at_company", y="salary")
sns.boxplot(data=df, x="left_company", y="salary")

Step 5 — Multivariate Analysis

sns.heatmap(df[["age","years_at_company","salary"]].corr(), annot=True)
sns.pairplot(df[["age","years_at_company","salary"]])

Step 6 — Spotting Anomalies

Step 7 — Forming Hypotheses

Chapter Attribution

Honest Scope Note

Hands-On Exercises