Project: a character-level language model

By now the block has a recurrent neural network (RNN) that reads texts of any length, a gradient that runs back through it in time, and two gated cells, the long short-term memory (LSTM) and the gated recurrent unit (GRU), that give it a memory able to survive distance. This lesson puts that machinery to work: building a model that, having read a text, can carry on writing it.

That kind of model has a name: a language model is a network that, given a stretch of text, predicts what comes next. At the character level (character tokenisation, from the lesson on tokenisationBlock 1 · Lesson 2Tokenisation: words, characters, subwordsBefore deciding which vector a unit gets, you have to decide what the unit is. That choice fixes the size of the vocabulary, the length of the sequences, and what it means for a word to be new.), that means the next character, whether a letter, a space or a full stop, and the whole task fits into that tiny question. It trains on a small corpus of a few thousand characters and is then asked to write, character by character, feeding itself what it produces. The text that comes out is the network talking to itself.

The network in the lesson on backpropagation through time (BPTT)Block 3 · Lesson 3Backpropagation through timeRecurrence shares one matrix across every position, and that changes what its gradient looks like. Backpropagation through time is the previous block's backward pass with the layers laid out along time: one error per step, carried from the end back to the start by the same W_hh, and a gradient that, because that matrix is reused at every position, is the sum of what each step asks of it. This lesson derives it in full and checks it, number by number, against a numerical probe. read a whole review and gave a single verdict: one output and one loss, hanging off the last state. A language model changes one thing and one thing only: it makes a prediction at every position. At step $t$, with the state $\mathbf{h}_t$ summarising everything read so far, it predicts character $t+1$; so a text of $T$ characters means $T$ predictions in a single pass, and its loss adds up all $T$ of them. That's what turns a text into a training set without anyone labelling it: the right answer at each position is the character at the next one, already written in the corpus.

The network is deliberately the simplest one in the block: the vanilla RNN from its own lessonBlock 3 · Lesson 2The vanilla RNNOne matrix applied at every position, and a state carried from each step to the next. That's enough to keep the length of the text out of every weight's shape: the same network reads two tokens or forty without padding or truncating anything, what it learns at position 2 works unchanged at position 8, and two sentences with the same tokens in a different order are no longer the same input., trained with BPTT as it stands, rather than the LSTM or GRU from the last two lessons. What's new here is the task, predicting the next character, and it's worth trying it out first on the cell whose backward pass has already been derived in two lines. With a gated cell, the backward pass would also have to cross every gate, and if something didn't add up, there'd be no telling whether the problem lay in the task or in the cell. What gets built around it (one output per position, the loss that adds them up, the sampling for writing) works the same for any of the three cells.

The honest thing is to say up front what it learns and what it doesn't. With a few thousand characters and a state of a few dozen coordinates, the network learns character statistics: where the spaces fall, that t is very often followed by h, that the, and and of are everywhere. It learns nothing about meaning or grammar. So what it generates sounds like English overheard from the far end of a noisy room: real short words, plausible long words that don't exist, and not a single sentence that tries to say anything. That isn't a defect this lesson sets out to fix: it's what a character model of this size learns, and naming it outright is half the lesson.

One output at every position

The recurrence is the one from the lesson on the vanilla RNNBlock 3 · Lesson 2The vanilla RNNOne matrix applied at every position, and a state carried from each step to the next. That's enough to keep the length of the text out of every weight's shape: the same network reads two tokens or forty without padding or truncating anything, what it learns at position 2 works unchanged at position 8, and two sentences with the same tokens in a different order are no longer the same input., without a single letter changed:

with $\mathbf{h}_0 = \mathbf{0}$. The first change is $\mathbf{x}_t$: since every token is a character, the input at position $t$ is that character's one-hot vector, $\mathbf{x}_t = \mathbf{o}_{c_t} \in \{0,1\}^{\lvert V \rvert}$, where $V$ is the set of distinct characters in the corpus. The vocabulary is made of characters now, not words, so $d_{\text{model}} = \lvert V \rvert$ and $\mathbf{W}_{xh} \in \mathbb{R}^{d_h \times \lvert V \rvert}$.

That one-hot vector brings with it the shortcut from the lesson on one-hot encodingBlock 1 · Lesson 4One-hot encoding and the curse of dimensionalityOne axis per vocabulary entry is the representation that asserts nothing about the words. It meets every demand made of it, and that is exactly why it lets you learn nothing.: multiplying a matrix by a one-hot vector picks out one of its columns, $\mathbf{W}_{xh}\mathbf{o}_{c} =$ column $c$ of $\mathbf{W}_{xh}$. So the columns of $\mathbf{W}_{xh}$ are the input vectors for each character, which the network learns as it goes.

The bigger change is the output. The lesson on the vanilla RNNBlock 3 · Lesson 2The vanilla RNNOne matrix applied at every position, and a state carried from each step to the next. That's enough to keep the length of the text out of every weight's shape: the same network reads two tokens or forty without padding or truncating anything, what it learns at position 2 works unchanged at position 8, and two sentences with the same tokens in a different order are no longer the same input. read a verdict off the last state; the language model reads one off every state, and it's a distribution over the next character:

with $\mathbf{W}_{hy} \in \mathbb{R}^{\lvert V \rvert \times d_h}$, one row for each character that might come next. The label is whatever the corpus actually put there: the one-hot vector of the next character, $\mathbf{y}_t = \mathbf{o}_{c_{t+1}}$. The loss at that position is the cross-entropy from the lesson on loss functionsBlock 2 · Lesson 5Loss functions: MSE and cross-entropyA loss sums up in one number how badly the network is doing, and between two candidates its derivative is what decides. Mean squared error hands over its smallest correction exactly where the network is wrong with the most confidence; cross-entropy, which comes from reading the output as a probability, leaves that derivative at the plain difference between what was predicted and the label., and the loss of the whole sequence adds one per position, averaged per character:

where $\hat{y}_{t,\,c_{t+1}}$ is the probability the network gave to the character that actually came. The $1/T$ takes the mean per position, the counterpart of the previous block's $1/B$ spreading $\mathcal{L}$ over the batch (except that here the "batch" is the $T$ positions of a single sequence).

The backward pass: every step receives two errors

The lesson on BPTTBlock 3 · Lesson 3Backpropagation through timeRecurrence shares one matrix across every position, and that changes what its gradient looks like. Backpropagation through time is the previous block's backward pass with the layers laid out along time: one error per step, carried from the end back to the start by the same W_hh, and a gradient that, because that matrix is reused at every position, is the sum of what each step asks of it. This lesson derives it in full and checks it, number by number, against a numerical probe. already did almost all of this calculation. There, only the last state touched the loss, so only $\boldsymbol{\delta}_T$ started from an output, and the interior steps received their error transported back from the next step. The language model changes exactly that: every position has its own output, so every step starts an error of its own and passes back the one arriving from further ahead.

It's still $\boldsymbol{\delta}_t = \nabla_{\mathbf{p}_t}\ell$, the error at the pre-activation. What changes is that the state $\mathbf{h}_t$ now reaches the loss along two paths (its own output at $t$, and step $t+1$), and the chain ruleBlock 2 · Lesson 7The chain rule, in earnestA hidden layer makes one and the same weight reach the loss by several paths at once, and the one-variable chain rule cannot cover that case. Within a path the derivatives multiply; across paths they add. Stacking those sums gives the Jacobian, and with it, differentiating a composition turns back into a matrix product. from the previous block adds them:

where the first term is the output error from the lesson on loss functionsBlock 2 · Lesson 5Loss functions: MSE and cross-entropyA loss sums up in one number how badly the network is doing, and between two candidates its derivative is what decides. Mean squared error hands over its smallest correction exactly where the network is wrong with the most confidence; cross-entropy, which comes from reading the output as a probability, leaves that derivative at the plain difference between what was predicted and the label. (softmax with cross-entropy delivers $\hat{\mathbf{y}}_t - \mathbf{y}_t$), distributed across the coordinates of the state by $\mathbf{W}_{hy}^{\top}$, and the second is the transport from the lesson on BPTTBlock 3 · Lesson 3Backpropagation through timeRecurrence shares one matrix across every position, and that changes what its gradient looks like. Backpropagation through time is the previous block's backward pass with the layers laid out along time: one error per step, carried from the end back to the start by the same W_hh, and a gradient that, because that matrix is reused at every position, is the sum of what each step asks of it. This lesson derives it in full and checks it, number by number, against a numerical probe., untouched. Mask with the slope of the $\tanh$ and you have the step's error:

with $\boldsymbol{\delta}_{T+1} = \mathbf{0}$, because there's no step after the last one. Put it next to the two equations from the lesson on BPTTBlock 3 · Lesson 3Backpropagation through timeRecurrence shares one matrix across every position, and that changes what its gradient looks like. Backpropagation through time is the previous block's backward pass with the layers laid out along time: one error per step, carried from the end back to the start by the same W_hh, and a gradient that, because that matrix is reused at every position, is the sum of what each step asks of it. This lesson derives it in full and checks it, number by number, against a numerical probe. and you'll see it's both of them joined together: the start from the output, which there belonged only to $\boldsymbol{\delta}_T$, and the transport, which there belonged only to the interior steps.

With each step's error in hand, the gradients are the ones from the lesson on BPTTBlock 3 · Lesson 3Backpropagation through timeRecurrence shares one matrix across every position, and that changes what its gradient looks like. Backpropagation through time is the previous block's backward pass with the layers laid out along time: one error per step, carried from the end back to the start by the same W_hh, and a gradient that, because that matrix is reused at every position, is the sum of what each step asks of it. This lesson derives it in full and checks it, number by number, against a numerical probe. plus the output layer's, all summed over the $T$ positions:

and likewise for the biases, summed. Everything inside the sums is a piece you've already derived; the only new thing is that $\mathbf{W}_{hy}$ is now shared across positions too (one output per step, not one at the end), so its gradient joins the sum over time instead of being a lone product.

Training and generating in NumPy

Three cells: the corpus, the training and the writing. Run them in order.

The first fetches the corpus (a few paragraphs about a seaside village, written for the course), lowercases it, strips out everything that isn't a letter, a space, a full stop or a comma, and turns it into a string of indices. Look at the vocabulary that's left, and at the last two lines, which sum up the whole model: what goes in is the text, and what comes out is the same text shifted by one position.

Out come 2,796 characters, 26 of them distinct (23 letters, since there's no j, x or z, plus the space and two punctuation marks), so $\lvert V \rvert = 26$: the same number as the alphabet, but not the same set. The last two lines are the entire training set in miniature: the village plus a space as input, and he village s as label, the same string moved one place along. That's where the almost 2,796 examples come from, one per position, with nothing labelled by hand.

The second cell is the complete model: the forward pass from the lesson on the vanilla RNNBlock 3 · Lesson 2The vanilla RNNOne matrix applied at every position, and a state carried from each step to the next. That's enough to keep the length of the text out of every weight's shape: the same network reads two tokens or forty without padding or truncating anything, what it learns at position 2 works unchanged at position 8, and two sentences with the same tokens in a different order are no longer the same input., which at every position produces one score per character for the softmax to turn into probabilities; the backward pass from the lesson on BPTTBlock 3 · Lesson 3Backpropagation through timeRecurrence shares one matrix across every position, and that changes what its gradient looks like. Backpropagation through time is the previous block's backward pass with the layers laid out along time: one error per step, carried from the end back to the start by the same W_hh, and a gradient that, because that matrix is reused at every position, is the sum of what each step asks of it. This lesson derives it in full and checks it, number by number, against a numerical probe., with the output's error added at every step; the gradient clipping from the lesson on the vanishing gradientBlock 3 · Lesson 4The vanishing gradientMultiplying by the same recurrent matrix at every step makes the gradient exponential in the distance it travels. What guarantees that it vanishes is γ·σ_max < 1, the steepest tanh slope times the largest singular value of W_hh; the spectral radius only sets the pace of the matrix on its own. Gradient clipping stops the explosion but can't rescue a gradient that has already vanished, and that's what will force a change of architecture.; and a descent step that updates the five groups of weights. It trains for 1,200 steps on 25-character chunks, each chunk an independent sequence starting from $\mathbf{h}_0 = \mathbf{0}$, as in the lesson on BPTTBlock 3 · Lesson 3Backpropagation through timeRecurrence shares one matrix across every position, and that changes what its gradient looks like. Backpropagation through time is the previous block's backward pass with the layers laid out along time: one error per step, carried from the end back to the start by the same W_hh, and a gradient that, because that matrix is reused at every position, is the sum of what each step asks of it. This lesson derives it in full and checks it, number by number, against a numerical probe.. Watch the two things it prints every few hundred steps: the loss per character, and a snippet of what the network would write at that point.

The loss starts at $3.26$, and that's no accident: it's $\ln 26$, the loss of a network that spreads its bets evenly across all 26 characters (an echo of the $\log 2$ from the previous block's classifier, with a 26-sided die instead of a coin). From there it falls smoothly to around $2.5$, and it would keep falling: the loop is cut short to fit within the ten seconds the browser allows a cell, because mediocre and on time beats good and never. The snippet on each line tells the same story from another angle: the first is noise, random characters; by the middle, there are spaces in the right places and some short words; by the end, it's broken English, real short words (of, the, age) threaded between long words that sound English without being English.

Chopping the text into chunks has a second cost: since every chunk starts from zero, the network never sees a dependency longer than 25 characters. For a model that only chases local statistics, that's a price worth paying; and the long-range dependency it gives up is exactly the kind the gated cells, the LSTMBlock 3 · Lesson 5LSTM: gated memoryA second state that moves forward by adding rather than by passing through a matrix, and three gates that decide what's kept, what's written and what comes out. Each gate arrives to fix a specific failure of the version before it, and the additive path changes the product that kills the gradient: instead of a power of the recurrent matrix, it becomes a product of numbers the network chooses coordinate by coordinate and can hold at one. and the GRUBlock 3 · Lesson 6GRU: the simplified versionAn update gate that ties keeping to writing (whatever isn't kept is exactly what gets written), a reset gate that lets the candidate ignore the past, and one state instead of two. It needs three quarters of the LSTM's weights, and the return path survives, because the direct path still multiplies by a number the network chooses and can hold close to one., knew how to keep.

The third cell uses the trained weights to write. Generating turns training inside out: instead of the real character that came next, the network receives the one it proposed itself. It starts at $\mathbf{h}_0 = \mathbf{0}$ with any character (a space, to begin a word), computes $\hat{\mathbf{y}}$, samples the next character from that distribution (draws one at random, each with the probability the network gives it), feeds it back as input, and repeats. The cell generates twice by sampling, and a third time by always taking the most probable character, so you can see the difference.

The first two lines differ from each other (that's the chance in the sampling), and both are broken English. The third behaves differently: always taking the most probable character is deterministic, and as soon as it falls into a high-probability run, it gets stuck repeating the same fragment over and over, in this case she. That's why generation samples instead of playing it safe: chance is what stops the network from chasing its own tail. Even so, neither of the sampled lines says anything meaningful. It can't: the network never saw a word as a unit, only characters, and it forgot anything more than 25 characters back. It's exactly what was promised, that the result would be bad, and watching it turn out that way is the lesson.

Test your intuition

Three questions: how many examples a text holds for this model, where each step of the backward pass gets its error from, and what training for much longer would fix (and what it wouldn't).

For the first time in the course, a network has written something. It's not much, and it's not good, but it's its own: none of those strings of characters appears in the corpus in that order, and still they look like English, because the network learned, letter by letter, what looking like English consists of. All of the block's machinery (reading any length, training through time, repairing the return path for the gradient) was there to get this far and switch something on.

But this model does only one thing: continue a piece of text in the same language it read. It can't take one sequence and return a different one (a question and its answer, a sentence in Spanish and the same sentence in English), because it handles one sequence, not two. Reading a whole sequence with one network and writing another with a second, tying them together through what the first one understood, is the subject of the next lesson, on sequence-to-sequence models: the encoder that summarises what was read and the decoder that writes from that summary, the last step before leaving the block.