Activation functions and non-linearity

The previous lesson, on the artificial neuron, left one piece still unchosen. The weights and the bias are the neuron itself (they fix the hyperplane, and with it everything it can ever tell apart), but between the pre-activation $z$ and what the neuron actually returns sits one more function, $\varphi$, that hasn't been chosen yet: the step filled that role by default, and only provisionally. Choosing it properly is what this lesson is about, and it's no cosmetic detail. Two things ride on $\varphi$: how much of the number the neuron computes survives, and what happens when you put one neuron behind another.

Start by watching what the step does when a weight moves. Take any neuron and nudge a single one of its weights from $1.0$ to $1.1$, to $1.2$, to $1.3$. The pre-activation tracks the movement exactly: it grows a little each time, passing through every value in between. But the output doesn't notice any of it. It sits at $0$ the whole time $z$ is negative, and the instant $z$ crosses zero, it jumps straight to $1$ and stays there. Whatever there was to learn about how far the weight had moved gets lost along the way, and always at the very same point.

Before writing down a single candidate, it helps to look at them side by side. The explorable draws each one over $[-5, 5]$, overlays its derivative as a dashed line, and shades in red the stretches where that derivative all but vanishes.

The derivative f′ button draws that dashed curve. The first four solid curves look a lot alike (all rising, all faster near zero), but their dashed curves have almost nothing in common: it's the derivative, not the function itself, that tells one candidate apart from another. And then there's the step, which shows exactly what it's missing: its dashed curve is a flat line pinned to zero, and the red covers the whole graph.

Why the activation cannot be a straight line

Start with the requirement that doesn't depend on any particular case. Chain two neurons together: the first looks at the input $\mathbf{x} \in \mathbb{R}^{d}$ and returns a number, and the second looks at that number and returns another. There's one neuron at each level, so the first level's weights form a vector and the second's is a single scalar; the superscript in parentheses marks the level, the same notation the layers will carry once each one holds several neurons:

with $\mathbf{w}^{(1)} \in \mathbb{R}^{d}$ and $b^{(1)}, w^{(2)}, b^{(2)} \in \mathbb{R}$. Now set $\varphi$ to the identity in both activations, $\varphi(z) = z$, and substitute the first expression into the second:

The right-hand side is a neuron: a single one, with weight vector $w^{(2)}\mathbf{w}^{(1)}$ and bias $w^{(2)}b^{(1)} + b^{(2)}$, and no input can tell it apart from the chain that produced it, since the two give the same number for every $\mathbf{x}$. The previous lesson already told you what that means: everything a neuron tells apart lives on its boundary, and that boundary is a hyperplane. So the chain of two separates with a hyperplane too. Chaining hasn't curved the boundary at all.

And there was nothing special about picking the identity. Try any straight line for $\varphi$, $\varphi(z) = \alpha z + \beta$ with $\alpha \neq 0$, and you'll find that $\alpha$ gets absorbed into the next level's weights and $\beta$ into its bias: what comes out the other side is, again, a pre-activation. Any straight line behaves the same way.

It helps to name this precisely. The expression $\mathbf{w}^{\top}\mathbf{x} + b$ isn't linear in the strict algebraic sense (a linear map sends zero to zero, and this one sends it to $b$ instead): it's affine. Composing affine functions always gives back an affine function, and that holds no matter how far you chain it: a hundred neurons in a row, each with a straight line for $\varphi$, add up to exactly one neuron. The property that has to break, and it can only break inside the activation, is the one this lesson takes its title from: non-linearity, though what actually gets broken is affineness.

The second requirement is easier to see in what happens to the step. Its derivative is zero everywhere it even exists,

and at $z = 0$ it doesn't even exist, because that's exactly where the function jumps. There's a debt worth naming out loud here: how a neuron actually finds its own weights from examples belongs to a lesson further on, the one on gradient descentComing up · Block 2 · Lesson 6Gradient descentThe gradient gathers one partial derivative per parameter and points, among all possible directions, at the one that raises the loss the most; subtracting a little of it each time is the entire method. How much "a little" turns out to be decides between converging, bouncing and diverging, and the whole sum can be worked out on a parabola.. All you need for now is what that method runs on: derivatives of $\varphi$. An activation whose derivative is zero everywhere has nothing to say about which direction to move the weights in, so the whole procedure grinds to a halt. The step passes the first requirement (it's about as far from a straight line as you can get) and fails the second completely.

Three candidates and their derivative

The three candidates that follow all meet both requirements, and I've written each one alongside its derivative already worked out: the rest of this block will lean on those derivatives far more than on the functions themselves.

The sigmoid already has its own reserved symbol in this course, $\sigma$:

continuous, increasing, with values in $(0, 1)$: it sends $z = 0$ to $\sigma(0) = 1/2$ and squeezes the extremes towards $0$ and $1$ without ever quite reaching either. Let's work out its derivative. Written as $\sigma(z) = \left(1 + e^{-z}\right)^{-1}$, the chain rule plus a convenient factorisation give

where

The sigmoid's derivative comes out in terms of its own value, with no fresh exponential needed, which is exactly what makes it cheap to evaluate thousands of times over.

The hyperbolic tangent traces the same shape over a different range of values:

Its values sit in $(-1, 1)$ and $\tanh(0) = 0$: it's centred on zero, which is the difference that actually matters compared with the sigmoid, whose output is always positive. Its derivative takes an equally convenient form,

and at the origin it equals $1$, four times higher than the sigmoid's own peak.

The third candidate is the youngest and the most widely used: ReLU, short for rectified linear unit. It's defined as $\text{ReLU}(z) = \max(0, z)$: it passes the positive part straight through and sends the negative part to zero, so it crushes nothing from above. Whenever $z$ is positive, its full magnitude reaches the output intact. Its derivative is

and at $z = 0$ it doesn't exist at all: the slope is $0$ coming from the left and $1$ coming from the right, and a function with a corner has no single tangent line at that point. This course takes it to be $0$ there, by convention rather than by any theorem, and the choice changes nothing that can actually be measured.

Where each one switches off

All three derivatives share one problem, and each one splits it up differently. Call saturation the region where the derivative of $\varphi$ is so close to zero that the input can move a long way while the output barely budges: that's what the red bands of the explorable mark out.

For the sigmoid, the arithmetic is straightforward. At $z = 5$, $\sigma(5) \approx 0.9933$, so its derivative there works out to $0.9933 \cdot 0.0067 \approx 0.0066$: almost forty times smaller than at the origin. The hyperbolic tangent switches off earlier and faster still ($1 - \tanh^{2}(5) \approx 1.8 \times 10^{-4}$, more than thirty times below the sigmoid at the same point), which is why its red bands start closer to the origin even though its derivative starts out four times higher. Starting higher and falling faster turn out to be entirely compatible.

ReLU breaks that pattern, and its half-graph red band doesn't contradict that at all. On the right, its derivative stays exactly $1$ no matter how far out you go: it never saturates there, and that's its whole appeal. On the left, it doesn't saturate either, since saturating means approaching zero: it's switched off outright, with the derivative sitting at $0$ over an entire half-axis. Out of that comes a defect with a name of its own, the dying ReLU: if a neuron's pre-activation stays negative for every example it ever sees, its derivative is zero on all of them, nothing ever moves it, and it never comes back.

That leaves the fourth button. GELU, short for Gaussian error linear unit, is the one used by the Transformers of Block 5; its definition calls on the cumulative distribution function of a normal distribution, and that's beyond the scope of this lesson. It looks much like ReLU except near the origin, where instead of folding into a sharp corner it curves smoothly.

The activations and their derivatives in NumPy

The cell defines the four functions and the three derivatives, then runs them through four checks. Run it and look, in this order, at whether the derivatives hold up against a numerical check, how much of each one survives far from the origin, what happens to the chain when the identity gets swapped for the sigmoid, and what the sigmoid does with the ten reviews from the previous lesson.

The first check earns the right to trust everything that follows. The central difference $\left(\varphi(z+h) - \varphi(z-h)\right) / 2h$ with $h = 10^{-5}$ approximates the derivative without knowing the formula at all, and all three agree with it to about $10^{-11}$. If you ever get an activation's derivative wrong, this catches it in two lines.

The second table puts actual numbers on saturation. The sigmoid's derivative drops from $2.50 \times 10^{-1}$ at the origin to $4.54 \times 10^{-5}$ at $z = 10$, four orders of magnitude; the hyperbolic tangent's covers eight over that same stretch. The ReLU row doesn't decay at all: $1$ across the last three columns, however far out you go. The $0$ in the first column is the convention above, written into code.

The third check settles the question without training a single thing. Every affine function $f$ satisfies the identity

no matter what $\mathbf{x}$ and $\mathbf{y}$ are (expand it out and you'll see the four biases cancel). With the identity as $\varphi$, the chain of two neurons gives $-6.66 \times 10^{-16}$: zero, as far as the computer's arithmetic can tell. With the sigmoid, it gives $-0.564$ instead. It's the exact same pair of neurons and the exact same inputs; the only thing that changed is the activation.

The last table circles back to the previous lesson: the same ten pre-activations from that review neuron, weights untouched, run through both the step and the sigmoid. The step hands back the same column as before; the sigmoid pulls apart what the step had lumped together. The review sitting right on the boundary drops from a confident $1$ to $0.500$, and the four negatives, which had all received the same flat $0$, now split into two distinct values: $0.269$ for the one sitting at $-1$, and $0.119$ for the three sitting at $-2$. The magnitude of $z$ is back, without touching a single weight.

Test your intuition

Three questions (a derivative, the collapse of the chain, and where each activation switches off), plus a coding challenge with ReLU.

There's still something worth saying about what $\varphi$ hasn't changed, before it starts to look like the sigmoid has solved more than it actually has. The decision boundary is exactly where it always was. Asking where $\sigma(z)$ crosses $0.5$ is the same as asking where $z$ crosses $0$, because $\sigma$ is increasing and $\sigma(0) = 1/2$: the hyperplane from the previous lesson survives every substitution this lesson has made, completely intact. What's changed is what the neuron reports about its decision, not the decision itself.

So stacking neurons can now build something a single one never could (without non-linearity, it couldn't even manage that much), but nobody has shown you a problem that actually forces you to stack them. There's a more uncomfortable question still missing: is there something a neuron, with any activation function at all, can never do, full stop? There is, and it fits in four points drawn on a sheet of paper: it's the next lesson, on XOR (exclusive or) and hidden layers.