Weather Forecaster: Predicting Tomorrow From dsproj1-3's Own Log

Data Science & ML Projects (Intermediate)

Chapter 5 · Weather Forecaster: Predicting Tomorrow From dsproj1-3's Own Log

Every model built so far in this course — churn, movies, digits — treated each row as independent. Shuffle them freely; nothing breaks. This chapter's data isn't like that. It's dsproj1-3's own growing weather log, and the entire point is that today depends on yesterday. A genuinely different problem shape, with its own genuinely different rules.

What We're Building

A next-day temperature forecaster, built by turning a sequential problem into an ordinary regression problem through feature engineering — teaching a model to predict today's temperature from the last few days' own values, then reusing that same model to genuinely forecast tomorrow.

Step 1: Loading the Log, In Order

import pandas as pd

df = pd.read_csv("weather_log.csv", parse_dates=["time"])
df = df.sort_values("time").reset_index(drop=True)
The rule that changes everything about this chapter
Every earlier chapter's own data could be shuffled without consequence — a customer row, a digit image, a movie rating, none of them cared about order. This dataset's own entire value comes from its order. Sorting by time explicitly, and never shuffling this DataFrame at any later step, is the one rule this whole chapter depends on.

Step 2: Resampling to Daily Granularity

daily = df.set_index("time").resample("D")["temperature"].mean().dropna().reset_index()

dsproj1-4's own .resample() tool, reused directly — dsproj1-3's own log is hourly; a next-day forecast is a daily-granularity problem.

Step 3: Engineering Lag Features

daily["temp_lag1"] = daily["temperature"].shift(1)
daily["temp_lag2"] = daily["temperature"].shift(2)
daily["temp_lag3"] = daily["temperature"].shift(3)

.shift(1) pulls each row's value from one position earlier into the current row — temp_lag1 on today's row literally is yesterday's temperature, now sitting as an input feature a regular regression model can use. This is the actual trick that turns a sequential forecasting problem into an ordinary tabular one.

Step 4: The Honest, Structural Missing Values This Creates

daily = daily.dropna().reset_index(drop=True)

The very first rows have no earlier days to pull a lag value from at all — temp_lag3 is undefined for day 1, 2, and 3. This is a structural byproduct of the lag-feature technique itself, not messy data the way dsproj1-2's own missing values were — nothing to fill or investigate, just a real, small, unavoidable cost of the method.

Step 5: The Split — Why Random Would Be Wrong Here

split_point = int(len(daily) * 0.8)
train = daily.iloc[:split_point]
test = daily.iloc[split_point:]
ml1-2's own train_test_split() would be a real mistake here
Every classification/regression project so far in this course correctly used a random split. Here, a random split would let the model train on a later day and get tested on an earlier one — effectively letting it "see the future" relative to what it's being tested on, producing a misleadingly good score that would never hold up in real forecasting. The correct split for any time-ordered problem is chronological: train on the earliest data, test only on data that comes strictly after it in time.

Step 6: Fitting the Model

from sklearn.linear_model import LinearRegression

features = ["temp_lag1", "temp_lag2", "temp_lag3"]
model = LinearRegression()
model.fit(train[features], train["temperature"])

ml1-3's own linear regression, applied here without modification — the model itself doesn't know or care that its inputs happen to be lagged values; that's entirely a property of how the features were engineered in Step 3.

Step 7: Evaluating — Against a Genuinely Honest Baseline

from sklearn.metrics import mean_absolute_error

predictions = model.predict(test[features])
model_mae = mean_absolute_error(test["temperature"], predictions)

naive_predictions = test["temp_lag1"]   # "tomorrow will be the same as today"
naive_mae = mean_absolute_error(test["temperature"], naive_predictions)

print(f"Model MAE: {model_mae:.2f}")
print(f"Naive baseline MAE: {naive_mae:.2f}")
The real standard for "does this model actually help"
ml1-4's own MAE metric, applied against a genuinely honest comparison: a "naive" forecast that just predicts tomorrow will match today. If the trained model's own MAE isn't meaningfully lower than this trivial baseline, it isn't actually adding predictive value — a real, standard sanity check in forecasting that a model beating "chance" or "average" (the usual bar elsewhere in this course) isn't automatically enough here.

The Complete Forecaster

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

df = pd.read_csv("weather_log.csv", parse_dates=["time"]).sort_values("time")
daily = df.set_index("time").resample("D")["temperature"].mean().dropna().reset_index()

for lag in (1, 2, 3):
    daily[f"temp_lag{lag}"] = daily["temperature"].shift(lag)
daily = daily.dropna().reset_index(drop=True)

split = int(len(daily) * 0.8)
train, test = daily.iloc[:split], daily.iloc[split:]
features = ["temp_lag1", "temp_lag2", "temp_lag3"]

model = LinearRegression().fit(train[features], train["temperature"])
model_mae = mean_absolute_error(test["temperature"], model.predict(test[features]))
naive_mae = mean_absolute_error(test["temperature"], test["temp_lag1"])

print(f"Model MAE: {model_mae:.2f} vs. naive baseline: {naive_mae:.2f}")

Order-dependent data

Never shuffled — the whole problem depends on strict chronological sequence.

Lag features via .shift()

Turns a sequential forecasting problem into an ordinary tabular one.

Chronological split

Train on earlier data, test only on strictly later data — never random here.

The naive baseline

"Tomorrow equals today" — the real bar a forecasting model must beat.

Extend This Project

Try these on your own:

  • Add precipitation as an extra lagged feature alongside temperature and see whether MAE improves.
  • Try a RandomForestRegressor in place of LinearRegression and compare against both the linear model and the naive baseline.
  • Extend the forecast horizon — predict two days ahead instead of one, and see how much the naive baseline's own advantage shrinks or grows.
  • Once you've taken nn1-8, try replacing this chapter's lag-feature approach with a real LSTM sequence model and compare MAE against both approaches here.

What's Next

Chapter 6: Named Entity Extractor — applying nlp1-7's own NER material to a real batch-document extraction tool, a structural sequel to Chapter 1's own whole-document sentiment classification.

Chapter 5 Quick Reference

  • A genuinely different problem shape — order-dependent, never shuffled, unlike every prior classification/regression project in this course
  • .shift() creates lag features, turning sequential forecasting into ordinary tabular regression
  • The train/test split must be chronological here — ml1-2's own default random split would leak future information
  • ml1-4's own MAE, evaluated against a genuinely honest "tomorrow equals today" baseline, not just a bare number
  • ml1-3's own linear regression, reused completely unmodified — only the feature engineering around it changed