Closures & Upvalues in a Bytecode VM

Writing a Compiler/Interpreter: Advanced

Chapter 7 · Closures & Upvalues in a Bytecode VM

Course 1's own tree-walking interpreter got closures for free — a WispFunction just kept a reference to the Environment active when it was declared, and Python's own garbage collector kept that object alive for as long as anything still pointed at it. This course's own VM has no such luxury. Chapter 4 made locals live directly on the shared value stack, and Chapter 6's own OP_RETURN deletes a function's locals the instant it returns. If an inner function captured one of those locals, what happens to it?

The Problem, Concretely

fun makeCounter() {
    var count = 0;
    fun increment() {
        count = count + 1;
        return count;
    }
    return increment;
}
var counter = makeCounter();
print counter();  // 1
print counter();  // 2
    

By the time counter() is called the first time, makeCounter has already returned. Its own call frame is gone. Per Chapter 6's own OP_RETURN, count's stack slot was deleted along with everything else makeCounter ever pushed. But increment's own body still says count = count + 1 — referring to a slot that, by any of the last six chapters' own rules, no longer exists.

Upvalues: an Indirection That Can Outlive Its Own Slot

The fix is an object that starts by pointing at the live stack slot, and later — the instant that slot is actually about to be destroyed — switches to holding its own private copy instead. Every read or write goes through this same object either way; the closure using it never has to know which state it's in.

class Upvalue:
    def __init__(self, stack, slot):
        self.stack = stack; self.slot = slot
        self.closed = False; self.value = None

    def get(self):
        return self.value if self.closed else self.stack[self.slot]

    def set(self, v):
        if self.closed: self.value = v
        else: self.stack[self.slot] = v

    def close(self):
        self.value = self.stack[self.slot]   # copy the current value out
        self.closed = True              # stop trusting the (soon-to-be-gone) slot
    

A function value now needs to carry its own captured upvalues alongside it, so WispFunction (the compiled code, shared across every call) is wrapped by a new Closure object (created fresh every time the fun statement actually runs, holding that particular call's own captured upvalues).

Compile-Time: Resolving a Name Across Function Boundaries

Chapter 6's visit_variable already checked "is this a local?" before falling back to a global. This chapter inserts a third possibility in between: "is this a local of an enclosing function?" Each Compiler now keeps a reference to the compiler for the function it's nested inside.

def resolve_upvalue(self, name):
    if self.enclosing is None:
        return None                          # the top-level script has nothing to enclose it
    local_slot = self.enclosing.resolve_local(name)
    if local_slot is not None:
        return self.add_upvalue(local_slot, is_local=True, name=name)
    outer_upvalue = self.enclosing.resolve_upvalue(name)   # walk OUTWARD, recursively
    if outer_upvalue is not None:
        return self.add_upvalue(outer_upvalue, is_local=False, name=name)
    return None
    

The is_local flag distinguishes two genuinely different situations: capturing a variable that's a plain local one level up (grab a stack slot directly), versus a variable an enclosing function itself already had to capture as an upvalue from somewhere further out (just forward that same upvalue along, unchanged). visit_variable/visit_assign now check resolve_local first, then resolve_upvalue, and only fall back to a global if neither finds anything — new opcodes OP_GET_UPVALUE/OP_SET_UPVALUE for the middle case.

OP_CLOSURE: Building the Real Runtime Object

A function statement no longer just pushes a bare WispFunction constant — it emits OP_CLOSURE, followed by one (is_local, index) pair per upvalue the function's own compiler discovered it needed.

elif instr == OP_CLOSURE:
    idx = code[frame.ip]; frame.ip += 1
    function = constants[idx]
    upvalues = []
    for _ in range(len(function.upvalue_descriptors)):
        is_local = code[frame.ip]; frame.ip += 1
        index    = code[frame.ip]; frame.ip += 1
        if is_local:
            upvalues.append(self.capture_upvalue(frame.base + index))   # a slot of THIS call
        else:
            upvalues.append(frame.closure.upvalues[index])              # forward one THIS closure already has
    self.stack.append(Closure(function, upvalues))
    

capture_upvalue checks whether an Upvalue already exists for that exact stack slot before creating a new one — necessary so that two closures capturing the same variable in the same scope end up sharing one object, not two independent ones that would silently drift apart.

Closing Upvalues at the Right Moment

Two different events can destroy a stack slot an open upvalue is still pointing at: the block it was declared in ending (Chapter 4's own end_scope), or the whole function returning (Chapter 6's own OP_RETURN). Both now close any upvalues affected, before the slot actually disappears.

# end_scope (Chapter 4), revised:
while self.locals and self.locals[-1][1] > self.scope_depth:
    self.chunk.write(OP_CLOSE_UPVALUE, line)   # was OP_POP
    self.locals.pop()

# OP_RETURN (Chapter 6), revised:
finished_frame = self.frames.pop()
self.close_upvalues(finished_frame.base)   # NEW -- close everything about to vanish
if not self.frames: return result
del self.stack[finished_frame.base - 1:]
    

OP_CLOSE_UPVALUE is safe to use unconditionally

Every local leaving scope gets this instruction now, whether or not a closure actually captured it — a deliberate simplification over tracking "was this specific local ever captured" at compile time. If nothing ever captured that slot, close_upvalues simply finds no matching entry and does nothing beyond the ordinary pop. The cost is a wasted dictionary lookup on locals nothing ever closed over; the benefit is not needing separate compile-time bookkeeping for "capturable" versus "ordinary" locals.

Verified: a Closure Outlives Its Own Creator

Verified directly — the counter keeps counting, three calls after makeCounter itself returned

counter() called three times in a row: ['1', '2', '3']. Directly inspecting the closure's own captured upvalue immediately after var counter = makeCounter(); completes — before counter() is ever called — confirms .closed == True and .value == 0.0: count's value was copied out and preserved at the exact moment makeCounter returned, exactly as designed.

Verified directly — two calls to makeCounter produce two genuinely independent counters

counterA(), counterA(), counterB(): ['1', '2', '1']. Each call to makeCounter declares its own count local, at its own stack position, captured by its own distinct Upvalue object — nothing is shared between the two closures just because they came from the same function declaration.

Verified: Two Closures Sharing the Same Live Variable

Verified directly — a setter and a getter, capturing the same local while it's still open, genuinely share it

fun makePair() { var shared = 0; fun setter(v) { shared = v; } fun getter() { return shared; } setter(42); return getter(); }

returns 42. setter and getter are two different closures, but capture_upvalue's own dedup-by-slot check means they share the identical Upvalue object — a write through one is visible through the other, because there was never a copy to begin with while makePair is still on the frame stack.

The Classic Bug: Closing Over a Loop's Own Counter

Course 1, Chapter 6 desugars for (var i = 0; i < 3; i = i + 1) { ... } into a var i declared once, outside a while loop — not redeclared fresh each iteration. If a closure inside the loop body captures i, every iteration's closure captures the same upvalue, because there's only ever one i.

Verified directly — three closures created across three iterations all report the same final value

Three closures, each capturing a shared loop counter i and stored for later, then all called after the loop finishes: ['3', '3', '3'] — not 0, 1, 2 as each iteration's own "current" value might suggest. This is the exact same well-known behavior real JavaScript's own var-based for loops have, and for the identical structural reason: one binding, shared, not reset per iteration.

This isn't a bug this chapter's own upvalue mechanism introduced — it's an honest, correct consequence of how for was desugared back in Course 1, faithfully reproduced. Wisp programmers hit the same trap real JavaScript programmers did before let existed.

The Fix: a Genuinely Fresh Local, Every Iteration

Verified directly — declaring a new local inside the loop's own body block gives each closure its own binding

Adding one line inside the loop body — var captured = i; — before each closure is created, then having the closures capture captured instead of i directly: ['0', '1', '2']. captured is declared fresh inside the body block on every single iteration; Chapter 4's own end_scope (now emitting OP_CLOSE_UPVALUE) closes the previous iteration's own captured before the next one is even declared, so each iteration's closure gets a genuinely distinct Upvalue, already closed with its own value, by the time the loop moves on.

Verified: Capture Two Function Levels Out

Verified directly — a variable captured through an intermediate function that never references it itself

outer declares x = 100; middle, declared inside outer, declares inner, which returns x directly — two levels out, skipping middle entirely. Result: 100. middle's own compiler never uses x itself, but resolve_upvalue's own recursive walk still registers an upvalue on middle (with is_local=False, forwarding outer's own upvalue index) purely so inner has something to capture — middle silently relays a variable it never touches.

Where This Connects

This chapter's finding	What it connects to
An `Upvalue` switches from pointing at a stack slot to holding its own copy	Chapter 6's own `OP_RETURN`, which deletes a function's locals immediately — the exact event this chapter's `close_upvalues` now races to get ahead of
Two closures capturing the same local share one `Upvalue` object, by dedup	Course 1, Chapter 7's own `WispFunction(declaration, closure)` — one shared `Environment` object achieved the same sharing for free in a tree-walking interpreter; here it takes a deliberate dedup check instead
Closures over a shared loop counter all see the same final value	Course 1, Chapter 6's own `for`-desugaring decision (one `var i`, not redeclared per iteration) — this chapter didn't create that behavior, it faithfully surfaced it
`OP_CLOSE_UPVALUE` replaces every scope-exit `OP_POP` for a local	Chapter 4's own `end_scope` — a direct revision, the same way Chapter 6 revised local addressing to be frame-relative

Hands-On Exercises

Exercise 1

Compile and run just fun makeCounter() {...} var counter = makeCounter(); (no calls to counter yet) using this chapter's own Compiler/VM. Inspect counter's own .upvalues[0] object directly and report its .closed and .value fields. Explain exactly which line of code caused .closed to become True, and why that happens before counter is ever actually called.

📄 View solution

Exercise 2

Compile and run fun makePair() { var shared = 0; fun setter(v) { shared = v; } fun getter() { return shared; } setter(42); return getter(); }. Confirm setter's own .upvalues[0] and getter's own .upvalues[0] are the exact same Python object (using is, not ==), and explain which specific line in capture_upvalue is responsible for that being true rather than each closure getting its own separate copy.

📄 View solution

Exercise 3

Reproduce this chapter's own loop-counter-capture bug (three closures over a shared for-loop counter, all returning the loop's final value), then reproduce this chapter's own fix (a fresh var captured = i; declared inside the loop body before each closure is created). Verify both results, and explain specifically why the fix's own captured variable gets a fresh Upvalue every iteration while the original i never does.

📄 View solution

Chapter 7 Quick Reference

Upvalue: open (reads/writes go to a live stack slot) or closed (reads/writes go to its own private copy) — the closure using it never has to know which
Compile time: resolve_upvalue walks outward through enclosing compilers; is_local distinguishes "capture a local directly" from "forward an upvalue an enclosing function already has"
OP_CLOSURE: builds the real runtime Closure, resolving each descriptor via capture_upvalue (dedup'd by slot) or by forwarding frame.closure.upvalues[index]
Verified: a counter closure keeps working three calls after its own creator returned; two separate calls to the same outer function produce genuinely independent counters
Verified: two different closures capturing the same still-open local share the identical Upvalue object, not just an equal value
Verified — the classic bug: three closures over a shared for-loop counter all report the loop's final value (['3','3','3']), reproducing real JavaScript's own well-known var-in-a-loop gotcha for the identical structural reason
Verified — the fix: a fresh local declared inside the loop body, captured instead of the loop counter itself, gives each closure its own value (['0','1','2'])
Next chapter: Garbage Collection — a real mark-and-sweep collector, needed now that closures can keep objects alive indefinitely