The context-vector bottleneck

The decoder from the previous lesson, on the encoder-decoder architecture, never reads the input sentence. It reads a vector. $\mathbf{c}$ has $d_h$ coordinates, that number is fixed when the network is built (before it has seen a single sentence), and it's the same for five tokens as for five hundred. That lesson flagged this for what it was: a bet on what fits in $d_h$ numbers, not something it had shown.

A bet can be lost in two very different ways, and telling them apart is the whole job of this lesson. A model can fail because it's undertrained, in which case more training fixes it. Or it can fail because what you ask of it doesn't fit where you ask it to fit, in which case no amount of training will help: the wall doesn't move. The lesson takes the two separately. First it counts two quantities and compares them. One is how many distinct values $\mathbf{c}$ can take. I'll call each of them a message, since it's the only thing that gets through from the encoder to the decoder. The other is how many distinct sentences the encoder can receive, where a sentence means any combination of tokens from the input vocabulary at a given length, not only the ones that appear in the data. For the decoder to reconstruct every sentence, each one needs a message of its own. Then the lesson measures what happens to a real model as the input grows longer, using two state widths and several training budgets.

The two quantities grow in incompatible ways, and the explorable lays one over the other. The green line doesn't move with the length: it shows the most distinct messages a vector of $d_h$ coordinates can encode, given that $q$ levels can be told apart in each coordinate (where $q$ comes from is the next section's business). That number knows nothing about the sentence it's summarising. The amber line counts the sentences of each length, and climbs a full rung with every token you add. Widen the state or raise the levels as far as you like, then look for the point where the two lines cross.

How many messages fit, and how many sentences there are

The encoder's state comes out of a $\tanh$, so each coordinate of $\mathbf{c}$ lives in the interval $(-1, 1)$. On paper that's a continuum, and anything fits in a continuum. In a model trained with gradients and read by another network, it doesn't. Let $q$ be the number of levels of one coordinate that can really be told apart: two values closer together than that reach the decoder as the same number. I choose $q$ myself: eight levels, sixteen, thirty-two, depending on how optimistic you want to be. What follows doesn't depend on which you pick.

With that convention the encoder has a finite number of distinct messages to send, and the sentences it has to tell apart are all those of length $T_x$ over the input vocabulary:

Look at where $T_x$ sits. In the first count it doesn't appear: the context vector is as big for a short sentence as for a long one, which is exactly the design decision of the previous lesson. In the second, it's in the exponent.

Now for the move that turns that asymmetry into an impossibility. Once $\mathbf{c}$ is fixed, everything the decoder writes is determined: it starts at $\mathbf{s}_0 = \mathbf{c}$, writes the most probable token at each step (the one with the largest coordinate in its output vector) and feeds that token back in. That's the previous lesson's greedy decoding. The complete output is therefore a function of $\mathbf{c}$, and a function can't return two different answers for the same argument. If there are fewer messages than sentences, two different sentences must share a message, so the decoder returns the same output for both and reconstructs at least one of them wrongly. Counting the sentences it can get right gives

The ceiling is $1$ (it says nothing) as long as the sentences fit, and from there on it's divided by $\lvert V_x \rvert$ with every new token. We find the point where it changes by taking logarithms in $\lvert V_x \rvert^{T_x} \le q^{d_h}$:

Plug in numbers of the kind real models use. A state of $d_h = 256$ coordinates, eight levels per coordinate and a subword vocabulary of $\lvert V_x \rvert = 32\,768$ entries give $T^{\star} = 51$ tokens; ten tokens further on, the ceiling above has already fallen to $10^{-44}$. And one word appears in none of the three formulas: training. Neither the number of iterations, nor the learning rate, nor the data enters the argument, because what we counted is the most different answers a mechanism can give, not how well it chooses among them.

A ceiling far above where it breaks

That count proves the wall exists. It doesn't say where the wall is, and the difference deserves spelling out. The next section trains encoder-decoders on a toy task: read a string of letters, drawn from only four of them (a, b, c and d), and write it backwards. The best of them has $d_h = 32$, and with $q = 8$ levels and $\lvert V_x \rvert = 4$ the count gives it a ceiling of $T^{\star} = 48$ tokens. Even so, it reverses only one five-letter string in three correctly.

The gap between those two figures has an explanation, and it's material from the previous block seen from a new angle. The gradient that teaches the encoder to keep token $j$ of the input starts at the loss term for position $i$ of the output, travels back through the decoder's steps as far as $\mathbf{s}_0 = \mathbf{c}$, and from there continues through the encoder's steps down to position $j$. There's no other route ($\mathbf{c}$ is the only point of contact between the two networks), and its length is

which is largest when $j = 1$, the first token of the input. The lesson on the vanishing gradientBlock 3 · Lesson 4The vanishing gradientMultiplying by the same recurrent matrix at every step makes the gradient exponential in the distance it travels. What guarantees that it vanishes is γ·σ_max < 1, the steepest tanh slope times the largest singular value of W_hh; the spectral radius only sets the pace of the matrix on its own. Gradient clipping stops the explosion but can't rescue a gradient that has already vanished, and that's what will force a change of architecture. describes what happens to a signal that crosses a route like that: each step multiplies it by $\mathbf{W}_{hh}^{\top}$ and by the $\tanh$ mask, and when $\gamma\,\sigma_{\max}$ (the steepest slope of the $\tanh$ times the largest singular value of $\mathbf{W}_{hh}$) is below one, the magnitude falls exponentially with distance. The first token of the sentence therefore receives the weakest signal of all.

So the encoder doesn't divide $\mathbf{c}$ among the positions the way an optimal code would. It fills it mostly with what it read last, because that's all it manages to learn to keep. The $48$ tokens of the count assume a perfect encoding; the handful of letters in the experiment is what you get from training a recurrence with gradient descent. Both are true, and they say different things: counting proves the wall exists, and training shows where it is.

Two state widths and three training budgets

The task is the one from the lesson on sequence-to-sequence modelsBlock 3 · Lesson 8Seq2seq: from sequence to sequenceTwo RNNs chained together: an encoder that reads the input sequence and compresses it into a single context vector, and a decoder that writes a new sequence from that vector. The pair trains end to end (the encoder learns only from the gradient the decoder hands back at c) on a toy task that runs in the browser: reversing strings of symbols. It works on short strings and degrades on long ones, because the whole input, whatever its length, has to fit into the same d_h numbers. That bottleneck is the doorway to the next block.: read a string of letters and write it backwards. It's deliberately impoverished (no grammar, no meaning, nothing to generalise), and that's what keeps the experiment clean: reversing a string takes nothing beyond remembering it, so every failure is a failure of memory.

The first cell builds a model of whatever width you ask for and defines its forward and backward passes, line for line the previous lesson's. The one thing to look for is that $\mathbf{c}$ is still the only link between the two networks: a single variable, marked in the code.

The second trains and measures: for each length, the percentage of strings that come out exactly right (a hundred and twenty new strings behind each number, generated without teacher forcing). The output has four rows. The first three are the same narrow state, $d_h = 8$, at three points in its training; the fourth has a state four times as wide and trains for half as long as the third. The test strings are the same in all four rows and the networks start from the same seed, so the only difference between rows is the one printed at the start of each.

Read the table in pairs of rows. From the first row to the second, training does exactly what you'd expect: in 750 more iterations, the two- and three-letter strings go from never coming out right to coming out right most of the time. From the second to the third, doubling the budget, the progress continues, but only in the short columns. The two-letter strings reach a perfect score and the three- and four-letter ones climb a little further, while the five-, six- and seven-letter ones stay at floor level, where they already were. That's the finding, and it's worth stating slowly: training buys accuracy on what fits, and nothing on what doesn't.

Now the fourth row, which trains for half as long as the third but with a state four times as wide: everything that had room to rise does, and the three long columns go from anecdotal to real (one five-letter string in three, against one in twelve). That answers the question from the start of the lesson, measured rather than guessed. The width of $\mathbf{c}$ and the training budget don't buy the same thing: training refines how well the vector is used, up to what the vector can hold, and only width moves the wall.

The most revealing thing, though, is how it gets things wrong, which is the previous section seen from the output's side. The lesson on sequence-to-sequence modelsBlock 3 · Lesson 8Seq2seq: from sequence to sequenceTwo RNNs chained together: an encoder that reads the input sequence and compresses it into a single context vector, and a decoder that writes a new sequence from that vector. The pair trains end to end (the encoder learns only from the gradient the decoder hands back at c) on a toy task that runs in the browser: reversing strings of symbols. It works on short strings and degrades on long ones, because the whole input, whatever its length, has to fit into the same d_h numbers. That bottleneck is the doorway to the next block. already let you see it on this same task: in a long string, the start of what the decoder writes (the end of what the encoder read, and so the most recent thing in $\mathbf{c}$) is usually right, and what gets scrambled is the end, which asks the decoder to remember the first token of the input. The worst-kept token is the one at the end of the longest route.

Test your intuition

Four questions: what more training fixes, how high the count's ceiling is, what widening the state buys, and which token of the input survives worst.

Both halves of the lesson point at the same place, and it isn't the size of $d_h$. The count weighs the messages that fit against the sentences there are because we agreed there would be a single message, and the route gets long because all the information passes through a single point. Widening $\mathbf{c}$ moves the ceiling a few tokens and leaves both of those statements untouched. The problem underneath isn't a small vector: it's the idea of summarising.

Put like that, something stands out. The encoder computed $\bar{\mathbf{h}}_1, \dots, \bar{\mathbf{h}}_{T_x}$, one state per input token, and the architecture throws away all but the last. The others were computed and then discarded. Getting them back, so that the decoder can look at the whole input instead of settling for the summary, is the job of the next lesson, on the idea of attention. Giving that backward look its formulas takes up the rest of the block.