Backpropagation through time

29 min read

The previous lesson, on the vanilla recurrent neural network (RNN), ended with a network that reads text of any length using two matrices and a loop, but that nobody has trained yet: every weight in it still holds the value it was given when the network was built. Training it means asking for the gradient of the loss with respect to those weights, and that question raises something the previous block never had to deal with. An RNN doesn't spread its work across one matrix per layer: $\mathbf{W}_{hh}$ is a single matrix that the loop uses at every position, so it's tangled up in the whole sequence.

How do you differentiate with respect to a weight that takes part in all $T$ steps at once? The previous block already has both halves of the answer. The chain rule says what happens when a weight reaches the loss along several paths; backpropagationBlock 2 · Lesson 8Backpropagation: the full derivationDifferentiating weight by weight forces a run over the network once per parameter, and those runs share almost everything. Backpropagation instead computes a single number per pre-activation (that neuron's error), in one pass from the end back to the beginning, and lets each weight read its own derivative as the product of that error with whatever it had in front of it. gets every derivative in a network in a single pass from the end to the start, without probing one weight at a time. What's missing is putting the two together inside a loop, and the result has a name of its own: backpropagation through time (BPTT). This lesson doesn't stop at stating it: it derives it in full and then checks, number by number, that its gradients match what a numerical probe measures.

Unrolling the loop lays the network out as a row of steps, and you can walk the backward pass along that row by hand. The loss sits at the end, hanging off the last state; to reach a weight, its influence has to retrace the route the forward pass took, from the last state to the one before, and from that one to the one before it, crossing the same $\mathbf{W}_{hh}$ at every hop. There's no other route, because there was no other route forward.

Step backwards one position at a time: each step drops its contribution into the same d_h by d_h accumulator. Raise T and check that more terms get added while the gradient's shape stays the same. At the last step, the contribution to the gradient of Whh is zero because h0 is zero; switch the accumulator to Wxh and you'll see that its contribution isn't.

The error that travels back, one step at a time

Fix one sequence, a single document, and do the derivation for that alone. A document is one whole example (one $\hat y$ , one label), so its loss is $\ell$ , the loss of a single example, as it has been throughout the previous block since the lesson on loss functionsBlock 2 · Lesson 5Loss functions: MSE and cross-entropyA loss sums up in one number how badly the network is doing, and between two candidates its derivative is what decides. Mean squared error hands over its smallest correction exactly where the network is wrong with the most confidence; cross-entropy, which comes from reading the output as a probability, leaves that derivative at the plain difference between what was predicted and the label.. The $T$ tokens the loop runs through aren't $T$ examples: they're positions inside this one document, the axis that plays the role of the layers here, and none of them is the batch. The batch is the other axis, the familiar one ( $B$ documents instead of one), and its mean, $\mathcal{L} = \frac{1}{B}\sum_i \ell_i$ , averages these per-document gradients one level further up, without touching anything that follows. The answer is read off the last state, as in the previous lesson:

\hat{y} = \sigma\left(\mathbf{W}_{hy}\mathbf{h}_T + \mathbf{b}_y\right),

so the recurrence's weights reach the loss in exactly one place: through $\mathbf{h}_T$ . The rest of the backward pass is a matter of reconstructing, step by step, how each state affects that last one.

First, give a name to something the previous lesson left unnamed. The pre-activation of step $t$ , the argument of the $\tanh$ , is

\mathbf{p}_t = \mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}_h \in \mathbb{R}^{d_h}, \qquad \mathbf{h}_t = \tanh\left(\mathbf{p}_t\right),

and call the gradient of the loss with respect to that pre-activation the error of step $t$ , $\boldsymbol{\delta}_t = \nabla_{\mathbf{p}_t}\,\ell \in \mathbb{R}^{d_h}$ . It's exactly the $\boldsymbol{\delta}^{(l)}$ from the lesson on backpropagationBlock 2 · Lesson 8Backpropagation: the full derivationDifferentiating weight by weight forces a run over the network once per parameter, and those runs share almost everything. Backpropagation instead computes a single number per pre-activation (that neuron's error), in one pass from the end back to the beginning, and lets each weight read its own derivative as the product of that error with whatever it had in front of it. (a layer's error, taken at its pre-activation), except that the index now counts steps in time rather than layers.

That error runs from the end back to the start, in two moves per step. The $\tanh$ acts coordinate by coordinate, so getting from the gradient at $\mathbf{h}_t$ to the gradient at $\mathbf{p}_t$ means multiplying by its slope. Using the identity $\tanh^{\prime} = 1 - \tanh^{2}$ from the lesson on activation functionsBlock 2 · Lesson 2Activation functions and non-linearityThe step passes along the sign of the pre-activation and drops everything else, and its derivative is zero wherever it even exists. Sigmoid, hyperbolic tangent and ReLU keep the magnitude instead and have a real derivative; none of them can be a straight line, because chaining affine functions together only ever gives back another affine function. together with $\mathbf{h}_t = \tanh(\mathbf{p}_t)$ , that slope is $\mathbf{1} - \mathbf{h}_t \odot \mathbf{h}_t$ , which you can read straight off the state the forward pass already stored, without keeping $\mathbf{p}_t$ around. That leaves the other factor, $\nabla_{\mathbf{h}_t}\ell$ , which splits into two cases depending on where the loss reaches $\mathbf{h}_t$ from: at the last step, only from the output layer; at every other step, only from the next one:

\boldsymbol{\delta}_T = \left(\left(\hat{y} - y\right)\mathbf{W}_{hy}^{\top}\right) \odot \left(\mathbf{1} - \mathbf{h}_T \odot \mathbf{h}_T\right),

\boldsymbol{\delta}_t = \left(\mathbf{W}_{hh}^{\top}\boldsymbol{\delta}_{t+1}\right) \odot \left(\mathbf{1} - \mathbf{h}_t \odot \mathbf{h}_t\right), \qquad t = T-1, \dots, 1.

The starting point, $\boldsymbol{\delta}_T$ , is the backward pass through the previous lesson's output layer: the sigmoid with cross-entropy delivers $\hat{y} - y$ , and $\mathbf{W}_{hy}^{\top}$ distributes it across the $d_h$ coordinates of the state. The recurrence handles everything else, and it's the two moves from the lesson on backpropagationBlock 2 · Lesson 8Backpropagation: the full derivationDifferentiating weight by weight forces a run over the network once per parameter, and those runs share almost everything. Backpropagation instead computes a single number per pre-activation (that neuron's error), in one pass from the end back to the beginning, and lets each weight read its own derivative as the product of that error with whatever it had in front of it. with nothing changed: it transports the error with $\mathbf{W}_{hh}^{\top}$ and masks it with the slope of the $\tanh$ . The only difference is that the transport always uses the same matrix. Hold on to that, because it's half of a later lesson, the one on the vanishing gradientComing up · Block 3 · Lesson 4The vanishing gradientMultiplying by the same recurrent matrix at every step makes the gradient exponential in the distance it travels. What guarantees that it vanishes is γ·σ_max < 1, the steepest tanh slope times the largest singular value of W_hh; the spectral radius only sets the pace of the matrix on its own. Gradient clipping stops the explosion but can't rescue a gradient that has already vanished, and that's what will force a change of architecture..

See the transport index by index

Here it is with no matrices in the way, for an interior step $t < T$ , where $\mathbf{h}_t$ reaches the loss only through the next step. Coordinate $j$ of $\nabla_{\mathbf{h}_t}\ell$ is the derivative of $\ell$ with respect to $h_{t,j}$ , and $h_{t,j}$ feeds into every one of the $d_h$ coordinates of the pre-activation $\mathbf{p}_{t+1}$ : it's the sum over paths from the lesson on the chain rule, with $d_h$ paths.

\frac{\partial \ell}{\partial h_{t,j}} = \sum_{i=1}^{d_h} \frac{\partial \ell}{\partial p_{t+1,i}}\,\frac{\partial p_{t+1,i}}{\partial h_{t,j}} = \sum_{i=1}^{d_h} \delta_{t+1,i}\,(\mathbf{W}_{hh})_{ij},

where the first factor is $\delta_{t+1,i}$ by the definition of the error, and the second comes from the fact that $p_{t+1,i} = \sum_k (\mathbf{W}_{hh})_{ik}\,h_{t,k} + \dots$ depends on $h_{t,j}$ only through the term $k = j$ , whose derivative is $(\mathbf{W}_{hh})_{ij}$ . That sum runs down column $j$ of $\mathbf{W}_{hh}$ , which is row $j$ of its transpose, in other words coordinate $j$ of $\mathbf{W}_{hh}^{\top}\boldsymbol{\delta}_{t+1}$ . So $\nabla_{\mathbf{h}_t}\ell = \mathbf{W}_{hh}^{\top}\boldsymbol{\delta}_{t+1}$ , and multiplying by the slope of the $\tanh$ gives $\boldsymbol{\delta}_t$ . At the last step, the same argument starts from the output layer instead of the next step, and that's where the other equation comes from.

The gradient of the shared weight is a sum

With one error per step in hand, the weight gradients follow without any further walking. This is where the sum the previous lesson promised turns up. Look at where $\mathbf{W}_{hh}$ takes part: in the pre-activation of every step, $\mathbf{p}_t = \mathbf{W}_{hh}\mathbf{h}_{t-1} + \dots$ , with $t$ running from $1$ to $T$ . It's a single weight reached along $T$ paths, one per position, and the chain rule from the previous block has already settled what to do about that: across paths, you add. Each path contributes what the lesson on backpropagationBlock 2 · Lesson 8Backpropagation: the full derivationDifferentiating weight by weight forces a run over the network once per parameter, and those runs share almost everything. Backpropagation instead computes a single number per pre-activation (that neuron's error), in one pass from the end back to the beginning, and lets each weight read its own derivative as the product of that error with whatever it had in front of it. gives for a pre-activation (the error times the transposed input), and the total is the sum of all $T$ contributions:

\nabla_{\mathbf{W}_{hh}}\ell = \sum_{t=1}^{T} \boldsymbol{\delta}_t\,\mathbf{h}_{t-1}^{\top}, \qquad \nabla_{\mathbf{W}_{xh}}\ell = \sum_{t=1}^{T} \boldsymbol{\delta}_t\,\mathbf{x}_t^{\top}, \qquad \nabla_{\mathbf{b}_h}\ell = \sum_{t=1}^{T} \boldsymbol{\delta}_t.

Put them next to the previous block's formulas and you'll find they're the same. There, a layer's gradient was $\boldsymbol{\delta}^{(l)}\left(\mathbf{h}^{(l-1)}\right)^{\top}$ , a single outer product; here it's that same outer product summed over the steps, because there each matrix appeared once, and here $\mathbf{W}_{hh}$ appears $T$ times. That sum is the only difference between backpropagation and backpropagation through time, and it's the exact price of sharing: one matrix, $T$ jobs, $T$ contributions to collect.

Two checks settle the account. The first is about shapes, the habit from the lesson on backpropagationBlock 2 · Lesson 8Backpropagation: the full derivationDifferentiating weight by weight forces a run over the network once per parameter, and those runs share almost everything. Backpropagation instead computes a single number per pre-activation (that neuron's error), in one pass from the end back to the beginning, and lets each weight read its own derivative as the product of that error with whatever it had in front of it.: each term of $\nabla_{\mathbf{W}_{hh}}\ell$ is $d_h \times d_h$ , and so is the sum, so the gradient has the shape of the matrix it corrects however long the sequence is. $T$ decides how many terms get added, not the size of the result. The second looks at the first term: in $\nabla_{\mathbf{W}_{hh}}\ell$ , the term for $t = 1$ is $\boldsymbol{\delta}_1\mathbf{h}_0^{\top}$ , and since $\mathbf{h}_0 = \mathbf{0}$ , it's the zero matrix. That fits with the previous lesson, where $\mathbf{h}_1 = \tanh\left(\mathbf{W}_{xh}\mathbf{x}_1 + \mathbf{b}_h\right)$ had no $\mathbf{W}_{hh}$ term: the first step never used that matrix, so it owes it no gradient. In $\nabla_{\mathbf{W}_{xh}}\ell$ , on the other hand, the $t = 1$ term does count, because $\mathbf{x}_1$ is a real token.

That leaves the output weights, $\mathbf{W}_{hy}$ and $\mathbf{b}_y$ , and they bring nothing new. They appear only once (the network reads out its answer once, at the end), so their gradient is an outer product with no sum, $\nabla_{\mathbf{W}_{hy}}\ell = \left(\hat{y} - y\right)\mathbf{h}_T^{\top}$ : the previous block's output layer, exactly as it was. The sum over time belongs to the three shared weights, and to nothing else.

BPTT in NumPy, against the numerical probe

This is the same check as in the lesson on backpropagationBlock 2 · Lesson 8Backpropagation: the full derivationDifferentiating weight by weight forces a run over the network once per parameter, and those runs share almost everything. Backpropagation instead computes a single number per pre-activation (that neuron's error), in one pass from the end back to the beginning, and lets each weight read its own derivative as the product of that error with whatever it had in front of it.: differentiate by hand, then compare every number against a probe that nudges each weight slightly and measures how the loss changes. The setup is a toy RNN with a fixed seed, a state of $d_h = 3$ and a sequence of $T = 5$ . Run it and look at two things: each step's contribution to $\nabla_{\mathbf{W}_{hh}}$ (the last one, for $t = 1$ , comes out as zero) and the final line.

import numpy as np

rng = np.random.default_rng(0)
d_model, d_h, T = 4, 3, 5
Wxh = rng.normal(size=(d_h, d_model)) * 0.5
Whh = rng.normal(size=(d_h, d_h)) * 0.5
bh = np.zeros(d_h)
Why = rng.normal(size=(1, d_h)) * 0.5             # scalar output: 1 x d_h
by = np.zeros(1)
X = rng.normal(size=(T, d_model)) * 0.5           # one row per position
y = 1.0                                            # the document's label


def forward():
  hs = [np.zeros(d_h)]                           # h_0 = 0
  for x in X:
      hs.append(np.tanh(Whh @ hs[-1] + Wxh @ x + bh))
  p = 1.0 / (1.0 + np.exp(-(Why @ hs[-1] + by)[0]))
  return hs, p, -(y * np.log(p) + (1 - y) * np.log(1 - p))


hs, p, loss = forward()
print("ŷ = %.4f   ℓ = %.4f" % (p, loss))

# Backwards: one δ per position, from the end to the start.
gWhh, gWxh, gbh = np.zeros_like(Whh), np.zeros_like(Wxh), np.zeros_like(bh)
dh = Why[0] * (p - y)                              # ∂ℓ/∂h_T, from the output layer
contributions = []
for t in range(T, 0, -1):
  delta = dh * (1 - hs[t] ** 2)                  # δ_t = (∂ℓ/∂h_t) ⊙ (1 - h_t²)
  gWhh += np.outer(delta, hs[t - 1])             # δ_t h_{t-1}ᵀ : what step t contributes
  gWxh += np.outer(delta, X[t - 1])
  gbh += delta
  contributions.append(np.linalg.norm(np.outer(delta, hs[t - 1])))
  dh = Whh.T @ delta                             # ∂ℓ/∂h_{t-1} : transport with Whhᵀ
gWhy, gby = np.outer([p - y], hs[T]), np.array([p - y])

print("each step's contribution to ∇Whh, from t=%d to t=1:" % T)
print("  ", np.round(contributions, 4), " <- the last one, t=1, is 0 because h_0 = 0")

# The probe: move each weight by hand and see how much the loss changes.
worst = 0.0
for P, G in [(Whh, gWhh), (Wxh, gWxh), (bh, gbh), (Why, gWhy), (by, gby)]:
  for k in np.ndindex(P.shape):
      v = P[k]
      P[k] = v + 1e-6; plus = forward()[2]
      P[k] = v - 1e-6; minus = forward()[2]
      P[k] = v
      worst = max(worst, abs((plus - minus) / 2e-6 - G[k]))
print("largest difference against the numerical probe: %.1e" % worst)

numpy

The first run downloads the Python interpreter (~15 MB). After that it stays in the browser cache and is reused across every lesson.

Check the output against the derivation. The list of contributions has five numbers, one per step, and the last is exactly zero: step $t = 1$ multiplies by $\mathbf{h}_0 = \mathbf{0}$ and asks nothing of $\mathbf{W}_{hh}$ , exactly as the algebra predicted. And the last line is the whole lesson: the gradients of all five groups of weights agree with the probe to within $10^{-10}$ , which is the probe's own error, not the algorithm's. The sum over time isn't an approximation or a workaround; it's the exact gradient, and the probe confirms it number by number.

Test your intuition

Four questions: the shape of the shared weight's gradient, where its sum comes from, what the backward recurrence looks like, and what the backward pass forces you to keep.

An RNN with a state of $d_h = 64$ coordinates reads a document of $T = 40$ tokens. How many numbers does $\nabla_{\mathbf{W}_{hh}}\ell$ hold?

A margin of ±0 is accepted.

In the previous block's networks, the gradient of each weight matrix was a single outer product. In the RNN, $\nabla_{\mathbf{W}_{hh}}\ell$ is a sum of $T$ of them. Where does that sum come from?

From

\mathbf{W}_{hh}

being a single matrix that the loop uses at all

T

positions: the loss reaches it along

T

paths, and the chain rule adds up the contributions from all of them.From the recurrence having

T

different

\mathbf{W}_{hh}

matrices, one per position, each of which has to be differentiated separately.From averaging the gradient over the

T

positions so that it doesn't depend on the length of the document.From the state

\mathbf{h}_t

accumulating the

t

previous positions, so that differentiating an accumulated quantity forces you to add up its terms.

Tick everything that is true of the backward recurrence $\boldsymbol{\delta}_t = \left(\mathbf{W}_{hh}^{\top}\boldsymbol{\delta}_{t+1}\right) \odot \left(\mathbf{1} - \mathbf{h}_t \odot \mathbf{h}_t\right)$ .

Select every correct option. This is graded all-or-nothing: there is no partial credit.

It transports the error with

\mathbf{W}_{hh}^{\top}

and masks it with the slope of the

\tanh

, the two moves of the previous block's backward pass; what's new is that the transport matrix is the same at every step.The mask

\mathbf{1} - \mathbf{h}_t \odot \mathbf{h}_t

\tanh^{\prime}

at that step, and it's read off the state

\mathbf{h}_t

that the forward pass already stored, with no need for the pre-activation.Each step transports with a different matrix, in the same way that the concatenated first layer had one block

\mathbf{W}^{(1)}_t

per position.The last step's error,

\boldsymbol{\delta}_T

, comes out of this same recurrence by setting

\boldsymbol{\delta}_{T+1} = \mathbf{0}

The backward pass runs through the steps from $T$ down to $1$ , using $\mathbf{h}_t$ and $\mathbf{h}_{t-1}$ at each one. What does that force you to do during the forward pass?

Keep the states

\mathbf{h}_1, \dots, \mathbf{h}_T

until the end, because each step's gradient needs them and the backward pass doesn't start until the last token has been read; memory grows with the length of the document.Nothing: the backward recurrence only uses

\mathbf{W}_{hh}

and

\mathbf{W}_{xh}

, which are never freed.Keep only

\mathbf{h}_T

, because the earlier states can be rebuilt by running the recurrence backwards from the last one.Compute each

\mathbf{h}_t

twice, once on the forward pass and once on the backward pass, to avoid using memory.

And then the whole backward pass, written by you: the piece the previous lesson's RNN was still missing before it could learn.

Write the recurrence's backward pass (the gradients of the shared weights), starting from what the forward pass left in storage.

bptt(hs, X, Whh, dhT) takes the list of states hs $= [\mathbf{h}_0, \dots, \mathbf{h}_T]$ with $\mathbf{h}_0 = \mathbf{0}$ ; the input X of shape $(T, d_{\text{model}})$ , whose row $t-1$ is $\mathbf{x}_t$ ; the recurrent matrix Whh; and dhT $= \nabla_{\mathbf{h}_T}\ell$ , the gradient the output layer passes to the last state.
It returns the triple (gWhh, gWxh, gbh), each one summed over the $T$ steps.

The function must not write into the list or the arrays it receives.

The first run downloads the Python interpreter (~15 MB); after that it stays in the browser cache. This challenge is much easier to solve on a physical keyboard: on a phone, read it and come back later.

The gradient is done, exact and verified, but it carries a warning inside it that we haven't read yet. The backward recurrence multiplies by $\mathbf{W}_{hh}^{\top}$ at every step (the same matrix, once per position), and it also masks with the slopes $1 - \mathbf{h}_t \odot \mathbf{h}_t$ , which the lesson on activation functionsBlock 2 · Lesson 2Activation functions and non-linearityThe step passes along the sign of the pre-activation and drops everything else, and its derivative is zero wherever it even exists. Sigmoid, hyperbolic tangent and ReLU keep the magnitude instead and have a real derivative; none of them can be a straight line, because chaining affine functions together only ever gives back another affine function. already showed switching off when the pre-activation $\mathbf{p}_t$ strays far from zero. An error that leaves position $T$ and has to reach position $1$ passes through that product dozens of times, and multiplying by one and the same matrix that many times does one of two things: it shrinks towards nothing, or it blows up.

Either outcome breaks exactly what the RNN was brought in to fix. If the gradient linking the first position to the loss has vanished by the time it gets there, the network can't learn that something at the start of the document mattered for the end: the long-range dependency, which was the whole point of having memory. How much it shrinks or grows, and why that comes down to a single property of $\mathbf{W}_{hh}$ , is the subject of the next lesson, on the vanishing gradient: we derive it carefully and then watch it plummet on a plot as the distance grows.

Was this helpful?