Linear Regression as Statistical Inference

Statistical Inference & Applied Statistics

Chapter 8 · Linear Regression as Statistical Inference

Linear Algebra Fundamentals' own Chapter 6 planted a forward reference: "fitting a straight line through a set of data points is solving a system of linear equations for the best-fit slope and intercept." This chapter is where that promise is finally kept — and it builds directly on Chapter 7's own correlation work, not separately from it.

Least Squares — Fitting the Best Line

Linear regression fits a line y = mx + b that best predicts y from x. "Best" means minimizing the sum of squared vertical distances between each actual point and the line — least squares.

Least squares formulas

m = Σ((x−x̄)(y−ȳ)) / Σ((x−x̄)²) b = ȳ − m·x̄

The Linear Algebra Fundamentals forward reference, paid off

That slope formula's numerator is exactly Chapter 7's own Pearson correlation numerator. These two least-squares formulas are, underneath, the solution to a small 2×2 system of linear equations — the "normal equations" — solved precisely the way Linear Algebra Fundamentals Chapter 6 described, using the exact same sum-of-products machinery Chapter 7 already introduced for correlation. Regression and correlation aren't two unrelated topics; regression is correlation's own numerator, repurposed to build a predictive line instead of just a strength score.

Worked Example: Fitting a Line to Chapter 7's Own Dataset

Reusing Chapter 7's exact CPU/response-time data:

Quantity	Value
Slope (m)	≈ 2.343
Intercept (b)	≈ 24.21
Fitted line	response ≈ 2.343 × CPU + 24.21

Residuals — What the Line Doesn't Capture

A residual is actual − predicted for each point — the vertical gap the line missed.

CPU	Actual	Predicted	Residual
40	120	117.93	+2.07
55	150	153.07	−3.07
60	165	164.79	+0.21
45	130	129.64	+0.36
70	190	188.21	+1.79
50	140	141.36	−1.36

Small, mixed-sign residuals across the board — the line fits this data closely, with no obvious pattern left unexplained.

R² — The Square of Chapter 7's Own r

R-squared

R² = r² — the proportion of variance in y "explained" by the linear relationship with x.

Chapter 7 already computed r ≈ 0.997 for this exact dataset — no new calculation needed: R² ≈ 0.997² ≈ 0.994. About 99.4% of the variation in response time across these six servers is explained by CPU usage alone in this linear model, leaving only about 0.6% unexplained.

Making a Prediction

Using the fitted line for a CPU value not in the original data, CPU = 65%: response ≈ 2.343 × 65 + 24.21 ≈ 176.5ms.

Extrapolation — a genuine danger

CPU = 65% sits comfortably inside the observed range (40–70%), so this prediction is reasonable. Plugging in CPU = 100% gives ≈ 258.5ms — but nothing in the actual data says the relationship stays linear that far outside the observed range. A server pushed to 100% CPU might degrade far more sharply (or hit a hard ceiling) than a straight line extrapolated from 40–70% data could ever predict. Trust a fitted line only within the range of x-values it was actually built from.

Chapter 7's own caution still applies in full

An R² = 0.994 is a genuinely excellent fit — and still says nothing about causation. High CPU usage causing slow responses remains only the most plausible of Chapter 7's own four explanations, not a proven one, regardless of how well the line fits.

Linear Regression in Code

def least_squares(x, y):
    n = len(x)
    xbar, ybar = sum(x)/n, sum(y)/n
    m = sum((x[i]-xbar)*(y[i]-ybar) for i in range(n)) / sum((xi-xbar)**2 for xi in x)
    b = ybar - m * xbar
    return m, b

cpu = [40, 55, 60, 45, 70, 50]
response = [120, 150, 165, 130, 190, 140]
m, b = least_squares(cpu, response)
print(m, b)  # 2.343, 24.21

def predict(x_new, m, b):
    return m * x_new + b

print(predict(65, m, b))  # 176.5

Hands-On Exercises

Exercise 1

Reusing Chapter 7's own sleep/bugs dataset (sleep = [7,5,8,4,6,7], bugs = [1,4,0,6,3,2]), fit a least-squares regression line, and use it to predict the number of bugs for 6.5 hours of sleep.

📄 View solution

Exercise 2

Chapter 7 computed r ≈ −0.985 for the sleep/bugs dataset. Using this chapter's own R² = r² relationship, compute R² and state, in plain terms, what percentage of the variation in bugs introduced is explained by hours of sleep in this model.

📄 View solution

Exercise 3

Using this chapter's own main worked example (CPU vs. response time, data ranging from 40% to 70% CPU), explain specifically why predicting response time at CPU = 5% is just as risky as the chapter's own CPU = 100% extrapolation example, even though 5% is a much smaller, seemingly more "reasonable" number.

📄 View solution

Chapter 8 Quick Reference

Least squares: m = Σ((x−x̄)(y−ȳ))/Σ((x−x̄)²), b = ȳ − m·x̄ — the same numerator as Chapter 7's Pearson r, repurposed to build a predictive line
This is literally Linear Algebra Fundamentals Chapter 6's own forward reference: fitting a line is solving a small system of linear equations (the normal equations)
Residual = actual − predicted, for each point
R² = r² — the proportion of variance in y explained by the linear relationship with x
Extrapolation risk: a fitted line is only trustworthy within the range of x-values actually observed
A high R² is still not proof of causation — Chapter 7's own correlation-vs-causation caution applies in full
Next chapter: Bayesian inference and updating beliefs