The vanilla RNN

The previous lesson, on why the multilayer perceptron (MLP) fails on sequences, measured two defects: what a long text loses to truncation, and what the network works out at one position but can't use at another. Both have the same cause, and it isn't the size of the network or the way it's trained. It's the input: that network never read a sequence, it was handed a vector. What follows is the fix, and it comes with a property nobody expects from a fix: the network that applies it is smaller than the one that fails.

It's called a recurrent neural network (RNN), and in its original form (the one everyone calls vanilla) it fits in two matrices and a loop. The hard part isn't how it works, which takes two lines, but why something so small manages what the big network couldn't. The whole answer lies in one thing it gives up: this network doesn't take in the sentence at a glance, it walks through it.

Walk through it yourself before reading on. In the explorable below, each step forward applies the same operation to one more position: the two matrices never change, and only the state coming in from the left does. This is also how you'll find it drawn in any book that covers it.

The picture invites one reading, and it's wrong: the boxes are not different layers. They're the same function drawn once per position, the way you'd draw the iterations of a for loop, which is why the labels repeat instead of being numbered. A network with one layer per position would have a separate set of weights for each; this one has a single set, however long the sequence.

A state rewritten at every position

As in the previous lesson, each position contributes the vector of its vocabulary entry, $\mathbf{x}_t = \mathbf{e}_{w_t} \in \mathbb{R}^{d_{\text{model}}}$, its own row of $\mathbf{E}$. What's new is the other vector, the one the network hands to itself from one step to the next: the hidden state $\mathbf{h}_t \in \mathbb{R}^{d_h}$, where $d_h$ is chosen when the network is built and is the only thing that decides how much that memory can hold. We'll also set $\mathbf{h}_0 = \mathbf{0}$, because before the first token has been read there's nothing to remember.

With that in place, a step is an ordinary layer that takes two inputs instead of one:

with $\mathbf{W}_{hh} \in \mathbb{R}^{d_h \times d_h}$ reading the previous state, $\mathbf{W}_{xh} \in \mathbb{R}^{d_h \times d_{\text{model}}}$ reading the current token, and $\mathbf{b}_h \in \mathbb{R}^{d_h}$. The hyperbolic tangent fills the role of $\varphi$ from the lesson on activation functionsBlock 2 · Lesson 2Activation functions and non-linearityThe step passes along the sign of the pre-activation and drops everything else, and its derivative is zero wherever it even exists. Sigmoid, hyperbolic tangent and ReLU keep the magnitude instead and have a real derivative; none of them can be a straight line, because chaining affine functions together only ever gives back another affine function., and any of the three covered there could fill it: the formula looks the same with each. What tips the balance here is something no layer in the previous block ever did: the output comes back in. $\mathbf{h}_t$ is the input to step $t + 1$, which in turn feeds step $t + 2$, so whatever the activation does to the state gets repeated once per token.

That rules out ReLU first, because it has no ceiling on the positive side. Picture a single coordinate with a recurrent weight of $2$ and inputs that don't push back: ReLU lets the whole state through, each iteration doubles it, and twenty iterations multiply it by more than a million. And all of that happens in the forward pass, before a single gradient has been computed. The hyperbolic tangent keeps every coordinate inside $(-1, 1)$, at $t = 1$ and at $t = 300$ alike, because its ceiling doesn't depend on what goes in. RNNs with ReLU do exist, but they need $\mathbf{W}_{hh}$ initialised carefully so that it doesn't stretch anything at the start; the $\tanh$ needs no such care.

The sigmoid is bounded too, but it loses to the $\tanh$ on the two differences that lesson pointed out. The first is the centre. With the sigmoid, every coordinate of the state would be positive: it could say how much, but never in which direction. And the $\mathbf{h}_0 = \mathbf{0}$ the reading starts from, which sits right at the centre of the $\tanh$'s range, would be an edge the sigmoid never actually reaches. The second is the slope at the origin: $1$ for the $\tanh$, $0.25$ for the sigmoid. When it's time to train this network, the error will travel from the last position back to the first, crossing that slope once per step, and with the sigmoid that factor would divide it by at least four every single time. That's why I write it by name rather than leaving it as $\varphi$: it's the activation the recurrence carries through the rest of the block.

Step $t = 1$ looks as if it needs a rule of its own, but it doesn't. With $\mathbf{h}_0 = \mathbf{0}$, the first term vanishes on its own:

That nesting answers the question of where the order is stored. Not in the weights (they're identical at all three positions), but in how many times each token has passed through the matrix $\mathbf{W}_{hh}$ by the time it reaches the end: in $\mathbf{h}_3$, the vector $\mathbf{x}_1$ has gone through it twice, $\mathbf{x}_2$ once and $\mathbf{x}_3$ not at all. Swap two tokens and you change those counts.

All that's left is to get an answer out, and there are $T$ states to choose from. A verdict for the whole document uses the last one, $\mathbf{h}_T$, because it's the only one that has read the entire text: the nesting above puts every token inside it, whereas any earlier $\mathbf{h}_t$ hasn't yet seen what comes after $t$. The answer comes out of one more layer sitting on top of it:

with $\mathbf{W}_{hy}$ of shape $1 \times d_h$ (one row per output, and here there's a single output) and a one-coordinate $\mathbf{b}_y$. The activation is different, and that's deliberate: $\sigma$ is the logistic sigmoid rather than the loop's $\tanh$, because what the network delivers is a probability, and the $\tanh$ returns numbers between $-1$ and $1$. This is the output layer of the previous block's review classifier, completely unchanged; the only difference is where its input comes from. That's why the drawing above leaves it out, and so does the weight count further down: the recurrence is the loop's two matrices, and this third one is a borrowed layer you already know how to build.

Now go back over the shapes of those two matrices and look for what's missing from both: $T$. $\mathbf{W}_{hh}$ is $d_h \times d_h$, $\mathbf{W}_{xh}$ is $d_h \times d_{\text{model}}$, and neither knows how many times it will be used. That leaves the two operations the fixed length made necessary with nothing to do: there's no $T_{\max}$ to truncate at, and no leftover positions to fill with the zero vector (the previous lesson's padding), because there are no extra positions. There are exactly as many as the document brings.

Underneath all this is a change of unit worth pausing on. In the concatenated MLP, one input to the network was a whole document: the first layer had as many columns as the entire document took up. Here, one input is a single token, because $\mathbf{W}_{xh}$ has $d_{\text{model}}$ columns and not one more, and at each step the network sees nothing but the vector at that position and the state it carries in. The length of the text hasn't vanished from the system: it has moved out of the shapes and into the number of times the loop runs. That shift changes everything, because a shape is fixed when the network is built and never touched again (which is what forced the truncation), whereas the number of iterations is decided by each document as it arrives.

One matrix for every position

Add up the weights in the recurrence's three pieces:

Set that beside the previous lesson's count, $d_1 \cdot T_{\max} \cdot d_{\text{model}}$. What matters isn't which one comes out smaller (that's a consequence) but which letters appear in each: $T_{\max}$ was in that one, and it's absent from this one. Reading documents ten times as long doesn't add a single weight.

And the saving is the least of it. What really changes is what each weight is for. With concatenation, the slice of the first layer that read position 2 and the one that read position 8 were two different matrices, adjusted by different examples, and the second stayed exactly as initialisation left it if the corpus never brought anything interesting that far along. In the recurrence there is one $\mathbf{W}_{xh}$, and the vectors from every position of every document pass through it. Whatever it learns to do with the vector of script, it applies wherever script turns up, because it has no way of knowing where that is.

Weight sharing is what makes an RNN an RNN. Take it away (a different matrix for each position) and what you're left with is the concatenated MLP under another name.

The catch comes with the gradient, and it's worth flagging now even though this lesson doesn't resolve it. $\mathbf{W}_{hh}$ doesn't take part in computing $\mathbf{h}_T$ once: it takes part $T$ times, as the nesting above shows. So the loss depends on that matrix along $T$ different paths, and its gradient will have to gather up every one of them.

Running the recurrence on two sentences of different length

Twelve vocabulary entries (this cell uses eight; the last four are for the next one), embeddings of $d_{\text{model}} = 4$ and a state of $d_h = 3$ coordinates. I draw the weights from a generator with a fixed seed, and they're untrained: what matters in this cell isn't the values that come out but how many times each matrix gets used. step is the recurrence transcribed line for line, and read applies it to a sentence, printing the state at each position.

Count the rows: five for the first sentence, ten for the second, and the same Whh and Wxh in all fifteen. There's no separate version of the network for short texts and long ones, and nothing has had to be truncated or padded: the loop runs as many times as the sentence needs.

The last two lines answer the previous lesson. the script sits at positions 1 and 2 in the first sentence and at 7 and 8 in the second, and the states it produces are almost identical: $(-0.602,\ -0.098,\ -0.527)$ against $(-0.632,\ -0.026,\ -0.540)$. Almost, not exactly, and both halves of that matter. They're similar because the weights reading those two tokens are the same weights, wherever the tokens happen to be; in the concatenated MLP there was no reason for them to be similar, and the previous lesson measured that they weren't similar at all. And they differ because the state each occurrence brought in wasn't the same, which is precisely the order information the bag of words had nowhere to keep.

You can isolate that difference, and the previous block left the perfect test case for it: the two sentences that closed the sentiment projectBlock 2 · Lesson 10Project: a sentiment classifierThe whole block's loop, now running over 240 film reviews with the previous block's bag of words feeding the input. The network gets all 180 it trains on right, and 86.7% of the 60 it has never read, and the eight it misses are not noise: four of them turn on a "not" the bag of words has no way to place.. They have the same tokens, repeated the same number of times, and they say opposite things.

The bag of words returns the same vector for both sentences, coordinate for coordinate, because it adds, and addition has no order. That's the input the project's classifier received, and it's why it gave both sentences $0.0002$.

The states narrate the reading step by step, and the table splits into two halves. The first four rows match digit for digit, because up to that point both sentences say the same thing, the film is not, and the network has nothing to tell apart yet. At the fifth row they diverge: that's the position where is not lands on boring in one sentence and on fun in the other. That project concluded that the not coordinate recorded how many times the word appeared, but not what it landed on. Here the not travels inside the state that reaches position 5, so the network reads the adjective with the negation right in front of it.

The final states land at $(-0.828,\ -0.077,\ -0.810)$ and $(-0.715,\ 0.167,\ -0.671)$. With untrained weights, the distance between them means nothing yet, and neither one means "positive" yet either. What the result does guarantee is that the output layer will receive two different vectors, and without that, no amount of training could ever give them different probabilities. With the bag, that condition had failed before anything even started. The recurrence has kept the best of both representations this block has been comparing: it tells orders apart, like concatenation, and it shares its weights, like the bag.

The last two lines put a number on the price, using the previous lesson's dimensions: $d_{\text{model}} = 100$ and a layer (or a state) of $128$ coordinates. Going from reading sentences to reading paragraphs multiplies the concatenated MLP's first layer by ten and leaves the recurrence exactly where it was, at $29\,312$ weights. The $3\,810\,688$ extra weights the MLP carries all start from their initialisation, and training has to adjust them with the same corpus as before.

Test your intuition

Three questions: how many weights the recurrence needs, why a position training never saw is no longer a problem, and what the formula does and doesn't say.

And then a full recurrence written by you, the very one you'll differentiate in the next lesson.

The network can read now, but it can't learn yet. Everything above is the forward pass: weights set by a random generator, not a single gradient, not one step of descent. And the gradient is where weight sharing stops being free. In the previous block's networks, each matrix appeared once on the path from the input to the loss, which is what made its gradient a single product; $\mathbf{W}_{hh}$ appears $T$ times.

What to do with those $T$ paths (add them up, walking the whole sequence backwards to collect them) is the subject of the next lesson, on backpropagation through time (BPTT). It's the previous block's chain rule with nothing changed, applied to a graph that's $T$ steps long instead of $L$ layers tall, and out of it comes the full calculation of $\nabla_{\mathbf{W}_{hh}}\mathcal{L}$, $\nabla_{\mathbf{W}_{xh}}\mathcal{L}$ and $\nabla_{\mathbf{b}_h}\mathcal{L}$.