Exercise 1: Why the Seq2Seq Bottleneck Gets Worse With Sentence Length — Possible Solution ==================================================================== WHY THE ENCODER'S FINAL HIDDEN STATE IS THE SAME IDEA AS nlp1-6's ------------------------------ Per this chapter, a seq2seq encoder is an LSTM that processes an entire source sentence and produces a single final hidden state — mechanically identical to nlp1-6's own sentiment-classification model, which also processed a whole sentence down to one final hidden state before handing it to a classifier head. The only difference is what happens to that summary afterward: nlp1-6 fed it to a sigmoid to produce one label; a seq2seq encoder hands it to a second LSTM (the decoder) as that decoder's own starting hidden state. In both cases, the entire sentence's information has already been compressed into one fixed-size vector before anything downstream ever sees it. WHY THIS COMPRESSION IS TOLERABLE FOR SHORT SEQUENCES ------------------------------ Per nn1-8's own mechanism, reused unchanged here, each hidden state is a function of the current input plus the accumulated history so far. For a short sentence, there are only a few steps of accumulation before the final hidden state is produced, so relatively little of the earlier information has had the chance to be diluted by everything computed afterward. WHY LONGER SENTENCES MAKE THE PROBLEM WORSE ------------------------------ As sentence length grows, the number of accumulation steps between the first word and the final hidden state grows too. Per this chapter, "early words in a long sentence get progressively diluted by every step that comes after them" — the same accumulating-history mechanism that let nlp1-6 preserve order now works against long-range fidelity, because the fixed-size final vector has a limited capacity, and a longer sentence is asking that same fixed capacity to represent proportionally more information. Something has to give, and what gives first is typically detail from early in the sentence. WHY THIS IS A REAL, MEASURED EFFECT RATHER THAN A THEORETICAL CONCERN ------------------------------ Per this chapter's own warn-box, translation quality measurably degrades as source-sentence length increases in a plain encoder-decoder setup — this wasn't a hypothetical worry researchers reasoned about in the abstract, it was an observed, documented pattern that directly motivated Bahdanau's 2014 attention mechanism as a fix. WHY THIS WORKS AS AN ANSWER ------------------------------ It identifies the specific structural parallel between nlp1-6's own final hidden state and a seq2seq encoder's final hidden state, explains using nn1-8's own accumulation mechanism why longer sequences place more strain on that same fixed-size representation, and connects this directly to the real, measured translation-quality degradation this chapter cites as the actual historical motivation for attention.