Tokenizing HTML: A Real, Forgiving Lexer
Building a Web Browser Engine: Parsing & the DOM
Chapter 2 · Tokenizing HTML: A Real, Forgiving Lexer
Chapter 1's depth-aware scan was a preview, built to answer one narrow question — how does an outer tag's own closing tag get found correctly. A real tokenizer has to answer a lot more: which tags never get a closing tag at all, what an attribute actually looks like, how a comment stays safe from the very characters that normally start and end a tag. This chapter builds that tokenizer for real — the thing every later chapter in this course actually consumes.
Five Kinds of Token
voidtag is its own kind, deliberately separate from opentag — a real, necessary distinction the rest of this chapter is mostly about.
Void Elements: Tags That Never Get a Closing Tag
A fixed, small set of HTML elements are defined to never have content and never get a closing tag at all — <br>, <img>, <input>, <hr>, and a dozen others. A tokenizer that doesn't know this list will wait forever for a </br> that a real HTML document is never going to contain.
tokenize("<br>") and tokenize("<br/>") produce byte-for-byte identical token lists — a single voidtag('br', {}) either way. Whether or not the source bothered to write the XML-style trailing slash, br was always going to be treated as content-free.
The Surprising Part: a Trailing Slash on an Ordinary Element Does Nothing
<div/> looks like it should close itself, the way it would in XML or JSX. Real HTML doesn't work that way — per the actual HTML5 spec, a trailing / on anything other than a void element (or an SVG/MathML element, out of this course's own scope) is simply ignored. The element opens normally and stays open.
tokenize("<div/><div/><div/>Hi</div></div></div>") produces three opentag('div', {}) tokens, then text('Hi'), then three closetag('div') tokens — never a single voidtag. Every one of those three / characters was silently thrown away. A real browser renders this exact markup with "Hi" genuinely nested three divs deep — not as three empty, self-closed boxes sitting side by side, which is what the JSX-trained eye expects on sight.
voidtag for <div/>, Chapter 3's own parser would never expect a matching </div>, and a real, valid document using this pattern would parse into a subtly wrong tree with no error raised anywhere. The fix lives entirely in one place: only treat the trailing slash as meaningful when the tag name is already in VOID_ELEMENTS.
Attributes: Three Shapes, One Parser
A single attribute string can hold double-quoted values, single-quoted values, unquoted values, and bare boolean attributes with no value at all — often all in the same tag.
tokenize('<div class="hello" id=\'world\' disabled>') produces attributes {'class': 'hello', 'id': 'world', 'disabled': None} — None specifically marking a boolean attribute, distinguishable from an attribute that was genuinely set to an empty string. Unquoted values work too: <input type=text value=42> parses to {'type': 'text', 'value': '42'}, scanning up to the next whitespace instead of a matching quote character.
Comments: Safe From the Characters That Normally Matter
The main scan loop treats < as the start of something structural and > as the end of a tag — but neither should mean anything inside a comment. Comments get their own dedicated scan, hunting specifically for the literal three-character sequence -->, ignoring everything else in between.
tokenize("<!-- a > b --><p>after</p>") produces one comment(' a > b ') token, correctly followed by a real p element — the internal > never terminated the comment early. The identical result holds for a stray <: "<!-- a < b -->" produces comment(' a < b '), confirming the comment scanner is genuinely delimiter-aware, not merely "looking for the next >" the way the tag scanner is.
Case-Insensitivity
tokenize("<DIV>Hi</DIV>") produces opentag('div', {}) and closetag('div') — both lowercased, matching real HTML's own case-insensitive tag names. Every later chapter — selector matching, the cascade, the default stylesheet — gets to assume a tag name is always already lowercase, because the tokenizer normalized it once, here, rather than every consumer needing to remember to do it themselves.
Where This Connects
| This chapter's finding | What it connects to |
|---|---|
A dedicated voidtag token, distinct from opentag | Chapter 3's own parser — a void element never gets pushed onto the open-element stack at all, sidestepping the entire "when does this close" question for the 14 tags that never need it answered |
A trailing / is ignored unless the tag is void | Chapter 3's own stack-based parser, which will nest <div/><p>...</p></div> exactly as deeply as this chapter's own token stream implies — the parser doesn't re-decide this; the tokenizer already did |
Attribute values default to None for boolean attributes | A forward reference to real CSS attribute selectors and form-control defaults — out of this course's own stated scope, but the distinction (present-with-no-value vs. genuinely absent) is preserved here regardless, for free |
| Tag names normalized to lowercase during tokenizing | Chapter 5's own selector matching — a CSS selector like div can be compared directly against a DOM node's own tag name with a plain string equality check, no case-folding needed at match time |
Hands-On Exercises
Tokenize "<div/><div/><div/>Hi</div></div></div>" — three "self-closing" divs in a row, matched by three real closing tags — using this chapter's own tokenize. Confirm the exact sequence of token kinds produced, and explain what a real browser would visually render for this markup (in terms of nesting, not exact pixels).
Tokenize "<input type=text value=42>" using this chapter's own tokenize. Confirm the resulting token's own kind and attrs, and explain why input produces a single token with no separate closing tag expected anywhere, even though this particular tag has real attributes on it.
Tokenize "<!-- a < b --><p>after</p>" — a comment containing a literal < rather than this chapter's own > example. Verify the comment's own text is captured correctly and that the following p element still tokenizes normally, then explain specifically which part of tokenize's own comment-handling branch is responsible for the < inside the comment never being treated as the start of a new tag.
Chapter 2 Quick Reference
- Five token kinds:
opentag,closetag,voidtag,text,comment - Void elements: a fixed 14-name set (
br,img,input, ...) that never get a closing tag, tokenized as their own dedicated kind - Verified — the real, surprising quirk: a trailing
/is ignored for any element that isn't void;<div/>opens normally and stays open, confirmed with three "self-closing" divs nesting three levels deep instead of closing - Attributes: double-quoted, single-quoted, unquoted, and boolean (value
None) all handled by one parser — verified on all four shapes at once - Comments: scanned for the literal
-->sequence specifically — verified safe from both a stray>and a stray<inside - Verified: tag names normalize to lowercase during tokenizing, so no later chapter needs to case-fold them again
- Next chapter: Parsing HTML into a DOM Tree — turning this token stream into the real, stack-based tree Chapter 1 only previewed