Exercise 1: BERT vs. T5 vs. GPT's Attention Directions, and the BiLSTM Parallel — Possible Solution ==================================================================== THE THREE ATTENTION DIRECTIONS, PER THIS CHAPTER'S OWN COMPARE-TABLE ------------------------------ BERT (encoder-only) uses bidirectional attention — "every token attends to every other token, both directions," with no restriction on which positions can see which. T5 (encoder-decoder) splits the two roles: its encoder is bidirectional exactly like BERT's, while its decoder is causal (each output token can only attend to itself, earlier output tokens, and the encoder's own output) — the decoder additionally cross-attends back to the encoder. GPT (decoder-only) uses causal attention throughout its entire single stack — every token, at every layer, can only attend to itself and tokens before it. WHAT nlp1-7's BIDIRECTIONAL LSTM ACTUALLY DID ------------------------------ Per nlp1-7, a standard forward-only LSTM can only use context that came before a given token, which caused real problems for words like "Washington" whose disambiguating information often arrives later in the sentence. A BiLSTM fixed this by running two separate LSTMs — one forward, one backward — and concatenating their hidden states, so each token's own final representation reflected context from both directions. WHY BERT'S BIDIRECTIONALITY IS THE SAME UNDERLYING IDEA, IN A DIFFERENT MECHANISM ------------------------------ Per this chapter, BERT's own attention "exists to let a token's own representation be informed by context on both sides, not just what came before" — the identical goal nlp1-7's BiLSTM was built to achieve. The mechanism differs completely: nlp1-7 achieved bidirectionality by running two separate sequential passes and concatenating their results, while BERT achieves it directly, since ordinary (non-causal) self- attention already lets every token attend to every other token in a single computation, with no need for two separate passes at all. WHY THIS COUNTS AS A GENUINE "TRANSFORMER-ERA COUNTERPART" RATHER THAN A COINCIDENCE ------------------------------ Both nlp1-7's BiLSTM and BERT's own bidirectional attention solve the identical problem — a token needing access to context on both sides, not just what preceded it — using whatever mechanism their respective architecture naturally offers. This is a case of the same underlying requirement (full-context awareness) being met differently by two different architectural eras: an explicit forward-plus-backward workaround for a sequential model that has no other way to see ahead, versus a direct, built-in capability for a model whose attention mechanism was never inherently directional to begin with. WHY THIS WORKS AS AN ANSWER ------------------------------ It states the specific attention-direction rule for each of the three architectures using this chapter's own compare-table, recaps nlp1-7's own BiLSTM mechanism and the problem it solved, and explains precisely why BERT's own bidirectional attention achieves the same goal through a structurally different but conceptually equivalent mechanism.