Every language implementation splits into a front end that understands source text and a back end that does something with the meaning it extracts — a lexer, then a parser, then evaluation or code generation. This course builds all of it, for a new toy language invented for this course: Wisp, a small, dynamically-typed scripting language with C-like syntax — variables, functions, closures, classes, and control flow. This chapter builds the very first stage: turning raw Wisp source text into a stream of tokens.

A Real, Working Lexer

Keywords vs. Identifiers

Maximal Munch: Why Multi-Character Operators Need Lookahead

Strings, Numbers, and a Real Error Case

Where This Sits in the Pipeline

Hands-On Exercises

Chapter 1 turned Wisp source text into a flat stream of tokens. A parser's own job is to recover the structure that flat stream implies — which operations happen before which, and how they nest. This chapter defines Wisp's expression grammar formally, then builds a real recursive descent parser that implements it correctly.

The Grammar, as a Precedence Ladder

This is a context-free grammar — each rule describes a category of expression purely in terms of what it's built from, with no reference to surrounding context. The key design choice is the order of the rules: each level only calls down into the level below it, never back up. That ordering is what encodes precedence directly into the grammar's own shape, before a single line of parser code exists.

Precedence, Verified

Associativity, Verified

Parentheses and Unary Minus

Where This Connects

Hands-On Exercises

Chapter 2's parser already produced a correctly-shaped tree — but it built that tree out of raw nested tuples like ('binary', 'PLUS', left, right). That works, but nothing stops a typo like 'PLSU' from silently producing a tuple that looks fine until something tries to read it. This chapter replaces those tuples with real, typed node classes, then confronts the maintenance problem that immediately creates: how do you add a new operation over a tree of several different node types without a giant, duplicated type-check in every single function that walks it? The answer is the Visitor pattern — introduced here from scratch, verified against the same "2 + 3 * 4" example this course has used since Chapter 2.

From Tuples to Typed Nodes

Each node is now a real, typed class instead of an anonymous tuple — Literal, Grouping, Unary, and Binary, matching the grammar rules Chapter 2 already defined. Every node also carries one extra method, accept(), whose only job is to call back into whatever visitor is walking the tree. Nothing about that method looks like it does much yet — its actual purpose only becomes clear once there's more than one thing that needs to walk the tree.

The Naive Way: One Type-Check Chain Per Operation

Before reaching for accept(), it's worth building the obvious first approach and seeing exactly where it breaks down. A function that evaluates a tree, and a separate function that prints one as a readable string, each need to ask "what kind of node is this?" — the obvious way is isinstance:

A New Node Type Arrives

Suppose a fifth node type shows up — Variable, a forward reference to Chapter 5's own variable lookups. Adding it means finding every function that walks the tree and adding a matching branch to each one's own type-check chain. It's easy to update one and genuinely forget the other — nothing forces you to touch both.

The Visitor Pattern: Double Dispatch

A note on where this sits relative to the rest of the site: this course's own Design Patterns course covers 16 of the classic Gang-of-Four patterns in depth, but names Visitor explicitly as one of the 7 left out of scope in its own opening chapter. So rather than a cross-reference, here is a compact, from-scratch introduction — because Visitor happens to be the standard, textbook answer to exactly the problem the last two sections just ran into.

The mechanism is called double dispatch. A visitor object implements one method per node type — visit_literal, visit_grouping, visit_unary, visit_binary. To evaluate a node, you don't ask "what type is this node" at all — you call node.accept(visitor), and the node's own accept() method (defined once, back when the node class was written) already knows which visit_ method to call back. Two dispatches: first to the node's own type via accept(), then to the visitor's matching method.

Adding an Operation Without Touching a Single Node Class

The real test: add a third operation — a NodeCounter that tallies how many nodes of each type appear in a tree — and see what each approach actually requires.

Failing Loud vs. Failing Silent

One more property falls out of the Visitor pattern for free, without anyone writing a single extra line to get it. Build an EvaluatorWithEnv that correctly implements visit_variable, and an AstPrinter that — realistically — hasn't been updated yet:

Where This Connects

Hands-On Exercises

Chapter 3's Evaluator could compute the value of a single expression — but a real Wisp program isn't one expression, it's a sequence of statements, some of which produce a value nobody reads and some of which exist purely to cause an effect, like printing something. This chapter turns that one-expression evaluator into a real interpreter: statement node classes with their own accept/visit dispatch, string and boolean literals the parser never handled before, comparison operators actually wired up to the evaluator, and two pieces of quiet, easy-to-get-wrong semantics — what counts as "true," and how a number should look when printed.

Statements Are Not Expressions

Every Expr node produces a value when evaluated. A statement doesn't — it's executed for its effect. Wisp gets two for now: an expression statement (evaluate something and throw the result away — this is what a bare 2 + 2; is) and a print statement (evaluate something, then actually output it).

Same shape as every Expr node from Chapter 3 — a small dataclass, an accept() method, double dispatch. The parser grows a second grammar layer above expressions: a statement() function that checks for the print keyword first, and a top-level parse_program() that keeps calling declaration() until it runs out of tokens, building a flat list of statements — an actual program, not just one expression.

Literals the Parser Never Handled Before

Chapter 1's lexer already produced STRING tokens and recognized true/false/nil as keywords — but Chapter 2 and 3's primary() only ever consumed NUMBER tokens. Printing anything interesting needs strings and booleans too, so primary() grows four new branches for them, each producing an ordinary Literal node wrapping a real Python str, bool, or None.

Truthiness: Not Python's

The unary ! operator needs an answer to "is this value true-ish?" for any Wisp value — a number, a string, nil. The tempting shortcut is Python's own bool()/not — except Python treats 0, 0.0, and "" as falsy, and Wisp deliberately doesn't.

Comparisons and a Real Equality Rule

Chapter 2's grammar already had a full precedence ladder for ==, !=, <, <=, >, >= — but Chapter 3's Evaluator only ever implemented the four arithmetic operators inside visit_binary. This chapter fills in the rest, and equality specifically needs its own helper rather than a bare Python ==.

Runtime Errors Instead of Leaking Python's Own

Feeding the wrong types into an arithmetic or comparison operator shouldn't crash the interpreter with a Python traceback a Wisp programmer has no way to make sense of. Each operator branch in visit_binary checks its operand types first and raises a dedicated WispRuntimeError with a message about the actual Wisp operation, before ever letting Python's own operators run on mismatched types.

Printing a Number Without Its Python Accent

Every Wisp number is a Python float under the hood, even literals that look like whole numbers — 3.0, not a separate integer type. Printed with Python's own str(), that's "3.0". A language with no separate int type shouldn't visibly leak that implementation detail every time a whole number gets printed.

Where This Connects

Hands-On Exercises

Every program so far in this course has been stateless — each statement ran with no memory of the ones before it. This chapter gives the interpreter somewhere to actually store values: an Environment. Declaring a variable, reading it back, reassigning it, and scoping it to a block all come down to one design question — how does an environment find a name that wasn't declared in its own immediate scope? Get that wrong, and the bug it produces is exactly the kind that looks fine until you nest two blocks.

Four New Node Types

Variables touch both halves of the AST built so far. Reading one is an expression — x needs a value the same way 2 + 3 does. Declaring and reassigning are statements and an expression, respectively — var x = 5; is a new kind of statement, while x = 5 is a new kind of expression that happens to also cause an effect. Blocks are statements too — a way to group statements and, critically, introduce a new scope.

Assignment is parsed at the very top of the expression grammar, above equality — parse_expression() now calls a new assignment() level first, which parses an ordinary expression, then checks whether it's followed by =. If the left side wasn't a Variable node, that's a parse error — 2 + 2 = 5; is nonsense, and this is where it gets caught.

Undefined Variables Fail Loudly

Reading a name that was never declared, and assigning to one, both need to raise a real, specific error — not a raw Python KeyError from a dict lookup, and not a silently-returned nil that would mask a typo as a legitimate empty value.

The Naive Environment: A Snapshot, Not a Chain

Blocks need their own scope — a variable declared inside { ... } shouldn't leak out. The obvious-looking first attempt: when entering a block, make a new environment by copying whatever the enclosing scope currently holds.

This looks reasonable, and even runs without error. The bug only shows up when a block reassigns a variable that belongs to an outer scope, then that outer scope is read again afterward.

The Fix: A Live Chain of Environments

The fix isn't to copy values at all — it's to keep a live reference to the enclosing environment, and walk up that reference chain whenever a name isn't found locally.

A block statement now executes its own statements against a brand-new Environment(self.env) — a child whose enclosing points straight at whatever environment was active before the block started — then restores the previous environment when the block ends. get and assign both fall through to self.enclosing when a name isn't local, recursing outward until either the name is found or the chain runs out.

Shadowing

A block declaring its own variable with the same name as an outer one shouldn't reassign the outer variable — it should create a brand-new binding that temporarily hides it. define() always writes into this environment's own values dict, never walking the chain the way assign() does — that asymmetry is exactly what makes shadowing work.

Where This Connects

Hands-On Exercises

Every program run through this interpreter so far has executed every statement exactly once, top to bottom. This chapter gives Wisp the ability to skip statements (if/else) and repeat them (while, for) — and along the way, a genuinely distinct kind of expression node, Logical, whose entire reason to exist is that it must sometimes decline to evaluate its own right-hand side at all. That turns out to matter for more than just and/or — this chapter closes with a real bug, found while writing it, in code this course has been running since Chapter 4.

if/else: A Straightforward New Statement

Nothing surprising here — the condition is evaluated once, and Chapter 4's own is_truthy() decides which branch (if either) actually runs. The interesting part of if isn't the statement itself; it's that the branch that isn't taken is never evaluated at all — stmt.then_branch.accept(self) is a real Python call that either happens or doesn't, not a value that gets computed either way and discarded. That's the same idea this chapter's real subject, Logical, needs to get right at the expression level.

while: The Same Idea, Repeated

The condition is re-evaluated fresh before every single iteration — not cached, not assumed. This is what makes a Wisp while loop able to see its own body's side effects: the body reassigns a variable via Assign (Chapter 5), and the very next condition check reads that new value straight out of the Environment.

for: Desugared Into while, Zero New Node Types

A C-style for (initializer; condition; increment) body doesn't need its own AST node or its own interpreter method at all. The parser can build it entirely out of pieces this chapter and Chapter 5 already have: an optional var declaration (or a bare expression statement) before a WhileStmt, whose own body is the original body followed by the increment, all wrapped in a BlockStmt so the loop variable stays scoped to the loop.

By the time this reaches the interpreter, it isn't a for loop anymore — it's a block containing a declaration and a while loop. Interpreter needs no new method whatsoever to run it; it was already capable of running exactly this shape.

Logical: A New Node, Because Binary Can't Do This

and and or look like they belong in Binary alongside + and < — but Binary's own visit_binary always evaluates both node.left.accept(self) and node.right.accept(self) before deciding anything. That's correct for arithmetic (you need both operands to add them) and wrong for logical operators, which are defined specifically so that the second operand is sometimes never even examined.

Notice what these branches return: not always true or false, but one of the two operand values themselves. That's deliberate — it's what makes a common default-value idiom possible.

Proving the Short-Circuit, Not Just Asserting It

"Short-circuit" is a claim about what doesn't run — which means the way to actually verify it is to put something that would visibly fail on the side that's supposed to be skipped, and confirm it never fails.

An Aside: The Same Mistake, Found Living in Chapter 4's Own Code

Writing the naive comparison above meant deliberately building code that evaluates something it doesn't need. That's worth checking for elsewhere in this course — and it turns up in a place with no relation to and/or at all: the arithmetic dispatch shown since Chapter 4.

Where This Connects

Hands-On Exercises

Everything built so far runs top to bottom, once. This chapter adds the ability to package up a sequence of statements, give it a name and parameters, and run it — possibly many times, possibly with different arguments each time, possibly calling itself. Functions turn out to need surprisingly little new machinery: one new callable class, one new exception type for return, and a careful answer to a question that's easy to get subtly wrong — when a function is declared inside another function, exactly which environment does it remember?

Three New Node Types

Call is parsed at a new precedence level between unary and primary — after parsing a primary expression, the parser checks for a following ( in a loop, so f(), and even a hypothetical f()(), both parse correctly without any special-casing.

WispFunction: A Real Callable

A Wisp function value isn't the FunctionStmt node itself — it's a wrapper object pairing that declaration with the environment that was active when the function was declared. That second piece is the closure, and it's the part this chapter spends most of its time on.

visit_function_stmt builds one of these when the fun statement executes: WispFunction(stmt, self.env) — self.env at that exact moment, not the global environment, not a copy. That single argument is the entire closure mechanism.

return: An Exception, Not a Value

A return can appear anywhere inside a function body — nested three ifs and a while loop deep, and it still needs to immediately stop everything and hand a value back to whoever called the function. A normal Python return from visit_return_stmt can't do that; it would only stop the current statement, not unwind out of however many nested blocks and loops are currently executing. Raising a Python exception can, because Python's own exception handling already unwinds through exactly that kind of nesting.

Recursion works the same way it does in Python, because a Wisp function call really is a Python function call. fun factorial(n) { if (n <= 1) { return 1; } return n * factorial(n - 1); } computing factorial(5) makes a real, recursive Python call to WispFunction.call() five levels deep.

Closures Capture a Live Environment, Not a Copy

"Closure" means the function remembers the environment it was declared in, and keeps seeing that environment's own live updates — not a frozen snapshot of what it contained at declaration time.

Each call to makeAdder gets its own fresh call environment (that's Chapter 5's own Environment, unchanged). adder is declared inside that call, so its closure is that specific call's environment — self.env at the moment fun adder(x) {...} executes, which is genuinely different for the makeAdder(5) call than for the makeAdder(10) call.

The Over-Capture Bug: One Environment, Reused

The bug isn't in what gets captured — it's in when the call environment gets created. WispFunction.call() above creates Environment(self.closure) fresh, every single call. A tempting, subtly wrong "optimization" is to create it once and reuse it:

This looks harmless in isolation — every individual call to a single WispFunction still runs correctly. The bug only appears when a function that itself declares and returns closures — like makeAdder — is called more than once, because now every one of those returned closures shares the exact same underlying call environment object.

The same live-capture mechanism also means a closure sees mutations made to a variable after the closure was declared, as long as they happen before the closure is called — closures over a mutable local, like a counter, work for exactly this reason.

The Call Stack Is Python's Own

This interpreter never built anything resembling a call stack — no explicit list of active calls, no frame objects. Every Wisp function call is a real, nested Python call (accept → visit_call → WispFunction.call → execute_block → accept → ...), so Wisp's own call depth is bounded by whatever bounds Python's.

Where This Connects

Hands-On Exercises

A class in Wisp is, structurally, not far from what Chapter 7 already built: a callable value (calling it constructs an instance, the same way calling a function runs its body) that owns a dictionary of methods, each of which is itself a WispFunction. The genuinely new problem this chapter has to solve is this — a method needs to know which specific instance it's currently running against, and getting that binding wrong is easy to do in a way that looks correct until two instances exist at once.

Three New Pieces: Classes, Instances, Properties

Fields aren't declared anywhere — WispInstance.fields starts empty, and a property is created the first time something is assigned to it via this.x = ... inside a method. Get (instance.field) and Set (instance.field = value) are two new expression nodes, parsed at the same precedence level as function calls — after a primary expression, the parser loops checking for either a following ( (a call) or a following . (a property access), so a.b.c() parses correctly without any special grammar rule for chaining.

this: Bound Fresh, Every Single Access

this isn't a keyword with special evaluation rules — it's parsed straight into Variable("this") in primary(), and resolved through the ordinary Environment chain like any other name. The entire mechanism is WispFunction.bind(), called every time a method is looked up on an instance via WispInstance.get():

Every property access that resolves to a method produces a brand-new WispFunction object, wrapping a brand-new environment, whose enclosing points back at the method's own original closure (the environment active when the class itself was declared) — with exactly one new binding, this, layered on top for this specific instance.

The Over-Mutation Bug: Binding in Place Instead of Creating Fresh

bind() looks like it's doing more allocation than necessary — a new environment and a new function object, every single time a method is touched. The tempting shortcut: define this directly into the method's own existing closure, and return the same function object unchanged.

self.closure here is the environment that was active when the class itself was declared — the same environment every instance's increment method shares, since there's only one increment declaration for the whole Counter class. Mutating it in place means every instance's own "binding" is actually the exact same shared slot, overwritten by whichever instance accessed the method most recently.

Single Inheritance: A Chain of find_method() Calls

class Dog < Animal { ... } stores Animal's own WispClass as Dog.superclass. Nothing about method lookup changes structurally — find_method() just doesn't stop at self.methods if the name isn't there; it asks the superclass to look too, recursively, for as many levels as the inheritance chain goes.

init Always Returns the Instance

A constructor's job is to set up an instance, not to compute a return value — so init is treated specially: even if its own body contains an explicit return statement, calling a class always yields the instance, never whatever init tried to return.

Where This Connects

Hands-On Exercises

Every chapter since Chapter 1 has raised an error of one kind or another — an unterminated string, a missing parenthesis, an undefined variable, a type mismatch — but each used whatever exception felt convenient at the time, with no shared shape and no line number attached to most of them. This chapter fixes that: one small error hierarchy, real line numbers threaded through the lexer and parser and onto every AST node that needs one, and a genuine Wisp-level stack trace for runtime errors — built by hand, since Chapter 7 already established that this interpreter has no call stack of its own to read one from.

Three Categories, One Shared Shape

Nothing about when each of these three fires has changed since the chapters that introduced them — the lexer still raises during tokenizing, the parser still raises during parsing, the interpreter still raises during evaluation. What's new is that all three now share one report() method and one consistent [line N] ErrorType: message format, instead of three unrelated ad-hoc exception shapes accumulated one chapter at a time.

Threading Line Numbers Through the Whole Pipeline

A line number has to originate somewhere real — the lexer's own scan position — and then survive being carried through every stage after that. Tokens grow a third field: (kind, lexeme, line), computed once per token as 1 + source.count('\n', 0, match_start). The parser then stamps that line onto every AST node it builds — Binary, Unary, Call, Variable, Get, Set, and FunctionStmt each grow a .line field, taken from whichever token triggered that node's construction (the operator for Binary, the opening ( for Call, the identifier itself for Variable).

Every place in the interpreter that already raised a WispRuntimeError — Chapter 4's type checks, Chapter 5's undefined-variable checks, Chapter 7's arity checks, Chapter 8's undefined-property checks — now passes node.line through, since the node being evaluated when the error is detected is always sitting right there in the same method.

Reachability: The Real Difference Between the Three Categories

The categories aren't just cosmetic labels — they genuinely behave differently, and the difference is about when the whole program has to be examined versus only the parts that actually run. Lexing and parsing both process the entire source file before a single statement executes. Interpretation only ever reaches the statements the program's own control flow actually visits.

A Real Wisp Stack Trace

Chapter 7 established that Wisp's own call "stack" is just Python's — there's no explicit list of active calls anywhere. That's fine for making recursion work, but it means there's nothing to read a Wisp-level stack trace from when something goes wrong three function calls deep. This chapter adds exactly that: an explicit call_stack list on the interpreter, pushed and popped around every function call.

The snapshot has to happen in the innermost call() frame that sees the error — the first one to run its own except WispRuntimeError block — because that's the only point where every frame from the outermost call down to the one that actually failed is still present in call_stack. The finally block pops this frame regardless, but only after the except block above it has already run and taken its snapshot; the if not hasattr(...) guard stops an outer frame from overwriting that snapshot with its own, now-truncated view as the exception continues propagating upward.

A Real Bug, Found While Writing This Chapter's Own Verification

Testing the stack trace against a recursive function surfaced a genuine gap: print 1 / 0; doesn't raise a WispRuntimeError at all.

Where This Connects

Hands-On Exercises

Nine chapters, each adding one working piece: a lexer, a parser, a tree of typed nodes, a tree-walking evaluator, an environment chain, control flow, functions and closures, classes and inheritance, and a real error-reporting layer with line numbers and stack traces. Nothing from any of those chapters gets replaced here — this capstone is one continuous, moderately complex Wisp program, run through the exact interpreter those nine chapters built, exercising closures, classes, and control flow together in a single working system rather than in nine separate, disconnected demonstrations.

The Program: A Task Queue, Built Without Arrays

Wisp never gained a native array or list type across this course — so the capstone's own data structure is a linked list, built entirely out of classes, the same way a real language without a builtin collection type would have to. This is deliberate: it's a genuine, useful demonstration of what classes alone can build, not a workaround apologized for.

UrgentTask overrides describe() exactly the way Chapter 8's own Dog < Animal example did. TaskQueue.forEach takes a function as its own argument — callback is called like any other value, because Chapter 7 established that functions in Wisp are ordinary, first-class values, not a special kind of name.

A Closure Factory, Reused Directly From Chapter 7

tally is a closure over two different things at once: checker, a parameter of countMatching, and count, a local variable it mutates on every matching task. It's also passed into queue.forEach as a plain value — three chapters' worth of machinery (Chapter 5's live environment chain, Chapter 7's closures, and Chapter 7's first-class functions) cooperating in four lines, with nothing here that any single earlier chapter didn't already establish on its own.

Running It

All tasks:
4
  - URGENT: Fix the leak
  - Read a book
  - URGENT: Submit tax filing
  - Water the plants
Critical tasks:
2
Priority >= 2:
3

Breaking It on Purpose

A capstone that only ever shows a program succeeding doesn't exercise Chapter 9's own contribution at all. Appending one deliberately broken line — passing nil where forEach expects a callable — triggers the real error-reporting pipeline this course spent an entire chapter building.

RUNTIME ERROR: [line 37] WispRuntimeError: can only call functions and classes
  at forEach() (called from line 82)
  at <script>

Where Each Piece Came From

What This Course Doesn't Cover

Wisp, as built across these ten chapters, has no array or list type (the capstone's own linked list is the workaround), no string formatting or number-to-string conversion (every print in this chapter avoided concatenating a number into a string, because Chapter 4's own + operator deliberately refuses to), no super.method() syntax for calling an overridden method's own parent implementation, no break/continue, no static methods or class-level fields, and no standard library beyond print. None of these are oversights discovered too late to fix — they're honest scope boundaries, each traceable to a specific chapter that could have covered them but chose a narrower, more teachable slice instead.

Where This Connects

Every one of these nine chapters' own components — the lexer, the AST, the Visitor pattern, environments, closures, classes, error handling — carries forward unchanged into Course 2: Writing a Compiler/Interpreter: Advanced. That course doesn't discard this interpreter; it rebuilds Wisp's own runtime as a bytecode compiler and a stack-based virtual machine, closing with a direct, measured performance comparison against the exact tree-walking interpreter finished here. The language stays the same. How it runs changes completely.