Why Build a Language? Lexical Analysis & Tokenization

Writing a Compiler/Interpreter: Fundamentals

Chapter 1 · Why Build a Language? Lexical Analysis & Tokenization

Every language implementation splits into a front end that understands source text and a back end that does something with the meaning it extracts — a lexer, then a parser, then evaluation or code generation. This course builds all of it, for a new toy language invented for this course: Wisp, a small, dynamically-typed scripting language with C-like syntax — variables, functions, closures, classes, and control flow. This chapter builds the very first stage: turning raw Wisp source text into a stream of tokens.

A Real, Working Lexer

KEYWORDS = {'var', 'fun', 'class', 'if', 'else', 'while', 'for',
            'return', 'true', 'false', 'nil', 'and', 'or', 'this', 'print'}

class Token:
    def __init__(self, kind, lexeme, line, col):
        self.kind = kind
        self.lexeme = lexeme
        self.line = line
        self.col = col
    

Verified directly — the lexer correctly tokenized a real, multi-line Wisp snippet with accurate line and column tracking

Tokenizing var greeting = "hello, " + name; followed by an if block: 19 tokens produced, each carrying its own exact line/col — including STRING('hello, ') at line 1, col 16, and PRINT('print') correctly identified on line 3, not line 1, after the multi-line source advanced the lexer's own line counter.

Keywords vs. Identifiers

Verified directly — a word starting with a keyword's own letters was correctly tokenized as a whole identifier, not truncated

Tokenizing classroom: a single IDENTIFIER('classroom') token, not a CLASS token followed by leftover characters. The lexer only classifies a word as a keyword after consuming the entire identifier — checking against the keyword set is the very last step, not something that happens character by character.

Maximal Munch: Why Multi-Character Operators Need Lookahead

two_char = source[i:i+2]
if two_char in ('==', '!=', '<=', '>='):
    tokens.append(Token(two_char, two_char, line, start_col))
    i += 2
    

Verified directly — a naive single-character-only lexer genuinely broke equality comparisons

A single-character-only version of the lexer, tokenizing "==" one character at a time: produces ['EQUAL', 'EQUAL'] — two separate assignment tokens, with no way for anything downstream to recover that this was meant as one equality check. The real lexer, checking two characters ahead first: correctly produces a single == token for a == b, while still correctly producing a single EQUAL token for the genuinely different a = b.

This is exactly what "maximal munch" means

At every position, the lexer always consumes the longest valid token it can — checking two-character operators before falling back to one-character ones. Tokenizing "a === b" confirms this directly: the result is == followed by a separate =, not three individual = tokens — the lexer greedily takes the first two characters as one token, then starts fresh from the third.

Strings, Numbers, and a Real Error Case

Verified directly — floats tokenize correctly, and an unterminated string is caught with a real, useful error

Tokenizing "3.14 + 2": NUMBER('3.14'), PLUS('+'), NUMBER('2') — the lexer correctly consumes a decimal point only when a digit follows it. Tokenizing var x = "never closed (no closing quote): raises "unterminated string starting at line 1" rather than silently consuming the rest of the file as string content or crashing with an unrelated Python exception.

A lexer that fails silently produces confusing errors two stages later

Without the explicit unterminated-string check, the lexer would keep scanning past the end of the source buffer looking for a closing quote that never arrives — either crashing with an unrelated index error, or (worse) silently treating everything after the opening quote, including real code, as string content. Catching it here, at the exact point the problem is knowable, is far more useful than any error a parser could produce from the resulting garbage token stream.

Where This Sits in the Pipeline

Stage	Input	Output
Lexer (this chapter)	Raw Wisp source text	A flat stream of tokens
Parser (Chapters 2-3)	The token stream	An abstract syntax tree
Tree-walking evaluator (Chapters 4+)	The AST	Program behavior

Hands-On Exercises

Exercise 1

Extend this chapter's own lexer to also recognize ! and != as separate tokens (currently, != is checked in the two-character map, but a bare ! has no single-character mapping and would raise an error). Verify tokenizing "!found" produces a BANG token followed by an identifier, and "a != b" still produces the existing != token correctly.

📄 View solution

Exercise 2

Tokenize a Wisp source string containing a string literal that itself spans multiple lines (a real, if unusual, case this chapter's own lexer already handles via its line-tracking inside the string-scanning loop). Verify the token's own reported starting line is correct, and that the lexer's own line counter has advanced correctly by the time tokenizing resumes after the string.

📄 View solution

Exercise 3

Tokenize the string "1..2" (two dots with no space) using this chapter's own lexer, and determine exactly what token sequence results. Explain whether this is a sensible tokenization for a hypothetical future "range" syntax, or whether it reveals a real gap in the number-scanning logic.

📄 View solution

Chapter 1 Quick Reference

The pipeline: lexer (source text → tokens) → parser (tokens → AST) → evaluator (AST → behavior)
Verified: the lexer correctly tokenized a real multi-line snippet with accurate line/column tracking on every token
Verified: "classroom" tokenized as one identifier, not a truncated CLASS keyword
Verified: a naive single-character-only lexer split "==" into two broken EQUAL tokens; the real lexer's two-character lookahead fixed it
Verified: an unterminated string raised a real, specific error instead of silently consuming the rest of the file
Maximal munch: always consume the longest valid token at each position
Next chapter: Grammars & Recursive Descent Parsing — turning this token stream into a structured tree