The vanishing gradient

The previous lesson, on backpropagation through time (BPTT), ended with a correct gradient and a loose end. Correct, because the numerical probe confirmed it entry by entry; a loose end, because computing it repeats a single move that nobody stopped to examine. To carry the error from one position back to the one before, you multiply it by the recurrent matrix $\mathbf{W}_{hh}$, and in a forty-token sequence that multiplication comes round almost forty times. This lesson measures what repeating the same multiplication that many times in a row does to the gradient, and finds the number in $\mathbf{W}_{hh}$ that puts a ceiling on the result.

This is no numerical curiosity: what's at stake is the one reason to prefer a recurrent neural network (RNN) over the network this block opened with. Take a review that starts I doubt this helps anyone and, thirty words later, ends with its rating. That opening I doubt flips the sign of everything after it, but the network can only learn the connection if the gradient computed at the last word survives the long trip to the first ones. If it fades along the way, the network never learns to pin the rating on those opening words, and the memory that defined it ends up, in practice, as a memory of the recent past only.

Before tackling the whole matrix, strip away coordinates until only one is left. With $d_h = 1$ the state is a number, $\mathbf{W}_{hh}$ is a number $w$, and carrying the error one step back means multiplying it by $w$. Repeating that step $d$ times multiplies it by $w^{d}$.

And $w^{d}$ has no middle ground. If $\lvert w \rvert < 1$ it sinks: with $w = 0.5$, forty steps leave the error at $0.5^{40} \approx 9 \times 10^{-13}$, twelve orders of magnitude below where it started. If $\lvert w \rvert > 1$ it shoots up: $1.5^{40} \approx 1.1 \times 10^{7}$. Only $\lvert w \rvert = 1$ keeps the error the same size, and that's a knife-edge a single point wide: stray a hair either way and you fall off one side or the other. With more than one coordinate, $w$ becomes a matrix, and its role splits between two numbers that the argument below keeps apart: one caps the gradient step by step, mask included; the other sets the long-run pace of the matrix multiplied by itself. The explorable shows the second, the spectral radius, for a recurrence with no mask (every slope of the $\tanh$ at $1$). Slide it to either side of $1$ and you'll see both halves of the phenomenon: below, the curve dives; above, it goes through the roof.

The product that links two positions

Go back to the previous lesson's backward recurrence and follow it for more than one step. There, each step's error came from the next one's in two moves, transport and mask:

The $\odot$ with the slope of the $\tanh$ amounts to multiplying by a diagonal matrix ($\text{diag}\!\left(\mathbf{1} - \mathbf{h}_t \odot \mathbf{h}_t\right)$, with those slopes on the diagonal and zeros everywhere else), so one step of the backward pass multiplies $\boldsymbol{\delta}_{t+1}$ by two matrices: first $\mathbf{W}_{hh}^{\top}$, then that diagonal. Chain the recurrence from the last step down to some distant step $k$ and the pair repeats, once per position:

The product has $T - k$ pairs, one for each step between $k$ and $T$ (the distance the error travels). And every pair contains the same $\mathbf{W}_{hh}^{\top}$: this isn't a different layer per position, the way the blocks $\mathbf{W}^{(1)}_t$ of the concatenated first layer were in the lesson on why the multilayer perceptron (MLP) fails on sequencesBlock 3 · Lesson 1Why the MLP fails on sequencesLining up the tokens' vectors builds order into the input, but at a price: a fixed length that forces you to truncate or pad, and a first layer carved into one block per position, each one trained only on what lands in its own position. A network that learns "not" in the first four positions scores 0.500 when "not" shows up in the fifth., but a single matrix raised, in effect, to the power of the distance. That's the move the previous lesson kept repeating without looking at it.

Why it decays or explodes

A product of matrices is hard to look at directly; its size isn't. Take norms on both sides and use the fact that the norm of a product is never bigger than the product of the norms:

That $\lVert \cdot \rVert$ is now measuring a matrix, not a vector. For any matrix $\mathbf{A}$ it's defined as

the largest factor by which $\mathbf{A}$ can stretch a vector's length, taking the worst case over every possible direction. That definition is what makes $\lVert \mathbf{A}\mathbf{v} \rVert \le \lVert \mathbf{A} \rVert \lVert \mathbf{v} \rVert$ true for any $\mathbf{v}$ (multiply both sides of the definition by $\lVert \mathbf{v} \rVert$), and applying it twice in a row gives $\lVert \mathbf{A}\mathbf{B} \rVert \le \lVert \mathbf{A} \rVert \lVert \mathbf{B} \rVert$: $\mathbf{B}$ stretches by at most $\lVert \mathbf{B} \rVert$, and $\mathbf{A}$ then stretches that result by at most a further $\lVert \mathbf{A} \rVert$.

Both factors in each pair have a ceiling. For a diagonal matrix, that maximum is easy to find: $\text{diag}(\mathbf{d})$ stretches each coordinate by its own entry, so the worst case is lining $\mathbf{v}$ up with the coordinate whose entry is largest in absolute value, and $\lVert \text{diag}(\mathbf{d}) \rVert = \max_i \lvert d_i \rvert$. Here each diagonal entry is a slope of the $\tanh$ (one per coordinate of $\mathbf{h}_t$): it never goes above $1$, and as soon as that coordinate saturates it gets close to $0$. Those are the red bands you already saw in the explorable in the lesson on activation functionsBlock 2 · Lesson 2Activation functions and non-linearityThe step passes along the sign of the pre-activation and drops everything else, and its derivative is zero wherever it even exists. Sigmoid, hyperbolic tangent and ReLU keep the magnitude instead and have a real derivative; none of them can be a straight line, because chaining affine functions together only ever gives back another affine function.. The largest of those slopes across every step from $k$ to $T - 1$ is therefore also $\le 1$; call it $\gamma$. The ceiling for $\mathbf{W}_{hh}^{\top}$ comes from the same definition, without the convenience of a diagonal, and the one new fact you need here is that a matrix stretches a vector's length by at most its largest singular value $\sigma_{\max}$, so multiplying by it $d$ times stretches it by at most $\sigma_{\max}^{d}$. Put the two bounds together over the $T - k$ pairs:

There's the exponent, and with it the one condition that holds whatever happens along the way. The error reaching position $k$ is at most the error at the end multiplied by $\gamma\,\sigma_{\max}$ raised to the distance. If $\gamma\,\sigma_{\max} < 1$, the gradient vanishes, with no escape: the ceiling plummets with the distance and the gradient can't rise above it, whatever the masks turn out to be. Since $\gamma \le 1$, $\sigma_{\max} < 1$ is enough to guarantee this on its own, and saturation only adds to the margin. The converse doesn't hold. If $\gamma\,\sigma_{\max} > 1$, the ceiling no longer holds the gradient down and it can explode, but nothing forces it to: that inequality is necessary for an explosion, not sufficient.

The bound is also usually too generous: a given vector rarely gets stretched by the maximum at every single step. What $\mathbf{W}_{hh}$ really does when multiplied by itself over the long run is measured by a different number, its spectral radius $\rho(\mathbf{W}_{hh})$ (the largest modulus among its eigenvalues), which never exceeds $\sigma_{\max}$. In a recurrence with no mask, every slope at $1$, the product is a power of $\mathbf{W}_{hh}^{\top}$, and there $\rho$ really does decide: below $1$ the gradient sinks, above $1$ it shoots up, and in between there's nothing but that knife-edge. It's the number the explorable let you move.

Once the masks get in the way, $\rho$ stops being a guarantee: the product alternates $\mathbf{W}_{hh}^{\top}$ with diagonals and is no longer the power of a single matrix. What still holds then is $\gamma\,\sigma_{\max} < 1$, which is why that's the real condition. You won't need to compute any eigenvalues in this lesson; what you do need to know is that $\rho$ exists, that it sets the pace of the matrix on its own, and that it never exceeds $\sigma_{\max}$.

Check it on a real matrix, without reducing it to a single coordinate: numpy.linalg has svd for $\sigma_{\max}$ and eig for the eigenvalues that give $\rho$.

For this $\mathbf{W}$, $\sigma_{\max} \approx 4.06$ but $\rho = 0.5$: the singular-value bound overestimates the pace of the power by a factor of eight. The table bears this out. At $d = 1$ the raw norm is $\sigma_{\max}$ (the two coincide by construction), and at $d = 5$ it's still above $1$ (the power can grow for several steps even though $\rho < 1$ guarantees it sinks in the long run). By $d = 40$ the raw norm is tiny and $\lVert \mathbf{W}^{d} \rVert^{1/d}$ has dropped below $0.58$, closing in on $\rho = 0.5$: actual numbers confirming what the argument above could only assert. And the two numbers answer different questions: the power on its own sinks, but with $\sigma_{\max} \approx 4.06$, this $\mathbf{W}$ guarantees nothing once the masks come in.

Seeing it and clipping it in NumPy

The first cell in this section boils everything down to the one-coordinate case and looks at the raw exponent; the second tackles the half of the problem that does have a cheap fix.

The scalar case isn't a toy: it's the formula above with $d_h = 1$, where $\sigma_{\max}$, $\rho$ and $\lvert w \rvert$ are all the same number. Run it and compare the middle row ($w = 1$) with the other two.

The first row sinks, the last shoots up and the middle one doesn't move, exactly as the start of the lesson said it would. With $w = 0.5$, the error arrives forty steps back at $9 \times 10^{-13}$ of its original size, and no gradient descent can learn from a signal that small. And that's the best case: as soon as the state saturates, the $\tanh$ mask pushes it lower still.

Explosion, on the other hand, has a straightforward answer. If the problem is that the gradient's norm grows with nothing to rein it in, give it a ceiling: fix a threshold $\kappa$ and, whenever $\lVert \mathbf{g} \rVert$ exceeds it, rescale the gradient to bring it back to exactly that size. This is gradient clipping:

The $\min$ with $1$ is the key: if the gradient is already small, the factor equals $1$ and leaves it alone; it only acts on a gradient that has gone over the limit, and it does so by multiplying it by a positive number, which changes its length but not its direction. Check it.

The normal gradient passes through untouched; the exploded one, with a norm of almost $30$, comes out clipped to exactly $5.00$, and its cosine with the original is $1$ (same direction, different size). Clipping rescues training when the gradient explodes, and it costs almost nothing. What it doesn't do, and this deserves saying out loud, is anything at all about vanishing: a gradient that has already shrunk to $10^{-13}$ has a tiny norm, the $\min$ equals $1$, and it stays exactly as it was.

Test your intuition

Four questions: what makes the gradient vanish or explode, what clipping fixes and what it doesn't, whether the $\tanh$ mask can save the day, and what's lost when it can't.

Clipping caps the explosion, but it leaves the other half of the problem where it was. A gradient that has been multiplied down towards zero doesn't recover by being multiplied by $1$, and no choice of $\mathbf{W}_{hh}$ fixes both at once: pushing $\sigma_{\max}$ below $1$ so the gradient can't explode is exactly what guarantees that it vanishes. The problem isn't the weights, it's the path: any path that multiplies by the same matrix at every step ends up at one of the two extremes.

The way out is to change the path. What's needed is a route along which the state moves from one step to the next by adding, rather than passing through $\mathbf{W}_{hh}$ yet again: across a sum, the gradient flows without being multiplied once per position, so it neither sinks nor shoots up. That additive path is the cell state of the long short-term memory (LSTM), and building it one gate at a time, each gate fixing a specific failure of the vanilla RNN, is the next lesson, on the LSTM and its gates.