Exercise 3: Why "Washington" Needs a Bidirectional LSTM — Possible Solution ==================================================================== WHY "WASHINGTON" IS GENUINELY AMBIGUOUS ON ITS OWN ------------------------------ Per this chapter, "Washington" could refer to a person's surname (e.g. George Washington) or a place (e.g. Washington, D.C., or Washington state). Nothing about the word itself resolves this — the ambiguity can only be resolved by looking at the rest of the sentence it appears in. WHY A PLAIN, FORWARD-ONLY LSTM (nn1-8) STRUGGLES HERE SPECIFICALLY ------------------------------ Per nn1-8's own mechanism, reused unchanged in nlp1-6 and this chapter, a standard LSTM processes a sequence strictly left to right — each hidden state depends only on the current input and everything that came BEFORE it. If "Washington" appears early in a sentence, e.g. "Washington signed the treaty," the LSTM's hidden state at the moment it processes "Washington" has no access at all to "signed the treaty," which comes later. It has to make its best guess about "Washington" using only whatever came before it in the sentence — which, at the very start of a sentence, may be nothing at all. A CONCRETE ILLUSTRATION ------------------------------ Compare "Washington signed the treaty" (a person, disambiguated by the verb "signed," which comes AFTER "Washington") against "Washington declared a snow emergency" (more likely the place, again disambiguated by what follows). A forward-only LSTM processing "Washington" as its very first token has identical information available at that exact step in both sentences — nothing yet exists to distinguish them — since the disambiguating word in each case only arrives later. WHAT A BIDIRECTIONAL LSTM ACTUALLY ADDS ------------------------------ Per this chapter, a BiLSTM runs two separate LSTMs over the same sequence — one processing it forward (left to right, exactly as in nn1-8), and a second processing it backward (right to left) — then concatenates both hidden states at every position. This means the representation for "Washington" is built from BOTH the (empty, in this case) context before it AND the full context after it, since the backward pass has already seen "signed the treaty" (or "declared a snow emergency") by the time it reaches "Washington" from the opposite direction. WHY THIS DIRECTLY ADDRESSES THE SPECIFIC FAILURE MODE ------------------------------ The problem was never that LSTMs can't use context to disambiguate words — nn1-8's whole mechanism is built around using context. The specific problem was that a forward-only LSTM's own notion of "context" only ever includes what came before a given word, which is structurally insufficient whenever the disambiguating information happens to come after it instead. A BiLSTM doesn't introduce a new capability so much as it removes an arbitrary directional restriction that had no reason to exist for a task like NER, where the correct tag for a token can depend on words appearing anywhere in the sentence, not just earlier ones. WHY THIS WORKS AS AN ANSWER ------------------------------ It identifies the specific structural limitation of a forward-only LSTM (hidden states depend only on preceding context), shows concretely why that limitation directly causes ambiguity for a word like "Washington" whose disambiguating context can arrive later in the sentence, and explains how a bidirectional LSTM's own forward-plus-backward concatenation removes exactly that limitation.