The artificial neuron

A vector can't decide anything on its own. Eight lessons of tokenising, counting and fitting have turned text into coordinates, which settles the input side, but something is still missing: the object that actually reads those coordinates and makes a decision. So far you only know its type ($f : \mathbb{R}^d \to \mathbb{R}^k$, set out back in the lesson on the problem of representing languageBlock 1 · Lesson 1The problem of representing languageA network operates only on real numbers. Text does not. This is the precise statement of the problem the whole block lives on. and left at that), and a type by itself computes nothing. This lesson builds the unit $f$ is made of, and it turns out that a single one is already enough to decide.

Here's a concrete case. Ten one-line movie reviews and a vocabulary of eight entries; each review becomes its bag of words, the vector that counts how many times each entry appears, exactly as in the lesson on the bag of wordsBlock 1 · Lesson 5Bag of words and TF-IDFSumming the one-hot vectors of a document gives a count vector where the function words rule. Weighting each coordinate by how rare the entry is in the rest of the corpus fixes that, and the logarithm that does it is no trick: it comes from measuring information.. So good movie i recommend it and bad movie very slow are two points in $\mathbb{R}^{8}$, and for each one you want the same thing: does the review speak well of the movie, or badly? Only two possible answers.

Each of those ten reviews is an example: a vector going in, paired with the answer you want out. That answer is its label, a number $y$ that's $1$ if the review is positive and $0$ if it's negative. A person set all ten labels by reading the reviews alone; they don't come from any calculation, and without them there would be no way to say an answer is wrong. Between the input and the answer, for now, there is nothing at all.

The simplest thing you could put in between is a vote. Each coordinate of the review's vector counts for or against the verdict depending on its value, and how strongly it counts is fixed by a number attached to that coordinate. Add up every contribution and out comes a single number, which then gets compared against a threshold: above it, one answer; below it, the other. And that's the whole object. There is nothing more to it.

That sum has a geometric picture worth seeing before writing anything down. With two coordinates per point, the most a drawing can show, the set of points where the sum exactly equals the threshold is a line, and what the object does is check which side of that line you landed on.

The line moves on its own because the explorable corrects its numbers every time a point lands on the wrong side; where those numbers actually come from is the business of much of this block, and for now you only need to watch it arrive. Once the two groups are separated, it arrives and stops. Notice how many different lines would work: if there's slack between the groups, a whole range of positions make zero errors, and the explorable keeps the first one it finds. Now drag a green point across into the other group: the notice below changes the instant no line can handle the split any more.

What a neuron computes

Start with the input: a vector $\mathbf{x} \in \mathbb{R}^{d}$, a column, like every vector in this course. A neuron is made of two things: a vector of weights $\mathbf{w} \in \mathbb{R}^{d}$, one per coordinate of the input, and a scalar $b \in \mathbb{R}$ called the bias. Together they compute a number,

the pre-activation, and the neuron feeds it through an activation function $\varphi$ to get

which is what the neuron actually returns. Both $z$ and $a$ are scalars: no matter what vector a neuron looks at, it always returns a single number. The type set out at the start of the course called for $k$ output numbers, and one neuron is enough to decide between two answers; the other $k - 1$ come from adding more neurons alongside it, each with its own weights, and that's where this block heads next.

$\varphi$ is still up for grabs, and for now I'll fix it by decree, using the oldest choice of all, the step:

With that, the neuron returns a $1$ or a $0$ and nothing in between. A neuron built around a step has had its own name since 1958, the perceptron, and it's the one that gives this block its title. There are other candidates for $\varphi$, and the choice matters, as you'll see shortly; the step is used here because it keeps the geometry in plain view.

It's worth reading $\mathbf{w}$ carefully when the input is a bag of words, because then each coordinate of $\mathbf{x}$ counts occurrences of a vocabulary entry, and each weight is the neuron's opinion of that entry: one number, positive or negative, nothing more. Because $z$ is a sum, those opinions stack up without interfering with each other (two occurrences of good contribute exactly twice what one does), and word order plays no part in any of it, since it was already thrown away by the representation. What the neuron adds on top of the bag of words is the verdict; whatever the bag had already lost stays lost.

The bias isn't anyone's opinion of a word. It's what the neuron thinks before it's read anything at all. It sets where the threshold falls, since comparing $z$ against zero is the same as comparing the sum of the opinions against $-b$. And in the picture, it's what frees the line from having to pass through the origin.

The word neuron is borrowed from biology, and it's best not to stretch the metaphor too far. A nerve cell integrates signals and fires once they cross a certain threshold, and that's where the resemblance ends. What you've read is a weighted sum followed by a comparison, and that's the whole of what the name is defending.

The boundary that separates, and how far from it you fall

$\varphi$ doesn't actually look at $z$, only at its sign. So everything the neuron can tell apart comes down to the set of points where $z$ changes sign, its decision boundary:

With $d = 2$ and $\mathbf{x} = (x_1, x_2)^{\top}$, solving that equation gives

which is exactly the line in the explorable above, as long as $w_2 \neq 0$; if it's zero, the line is vertical, and the original equation describes it equally well. In $\mathbb{R}^{3}$ the same set is a plane, and in $\mathbb{R}^{d}$ it's a hyperplane: an object of dimension $d - 1$ that splits the space into two halves. For the reviews below, $d = 8$; with a real vocabulary it would be tens of thousands. You can't draw the hyperplane at that point, but it's the same object all the same.

The weights don't only place that boundary, they're perpendicular to it. Take any two points $\mathbf{x}$ and $\mathbf{x}^{\prime}$ that lie on it. Both make the equation equal zero, so subtracting one from the other cancels the bias and leaves

where $\mathbf{x} - \mathbf{x}^{\prime}$ is any direction lying within the boundary. Its dot product with $\mathbf{w}$ is always zero, which means $\mathbf{w}$ points perpendicular to the boundary, towards the side where $z$ is positive.

The bias is what shifts it. With $b = 0$, the zero vector satisfies the equation, so the boundary is forced through the origin and can't pass anywhere else; the moment $b \neq 0$, it moves away. Without a bias, the answer depends only on the direction of $\mathbf{x}$; with one, it also depends on where $\mathbf{x}$ actually is.

What's still missing is what makes $z$ worth more than its sign alone. The signed distance from any point to the boundary, positive on one side and negative on the other, is

which means $z$ is that distance, scaled by $\lVert \mathbf{w} \rVert$. Its magnitude tells you how much margin the decision had, and the step keeps only the sign, throwing the rest away.

A neuron that scores reviews in NumPy

The cell builds the whole case from scratch: ten reviews, eight vocabulary entries, the bag-of-words vector for each, and a neuron on top. I've set the weights by hand ($+1$ for entries that count in favour, $-1$ for those that count against, $0$ for the ones with no opinion), since this lesson doesn't yet give you a way to choose them. Run it and pay attention to three things: the column of $z$ values, what happens when you multiply the weights by ten, and what happens when you nudge the bias down by half a point.

Nine out of ten correct, and the one it misses is exactly the one you'd expect. good movie but slow adds $+1$ for good and $-1$ for slow, so its $z$ comes out to exactly $0$, right on the boundary. The step breaks the tie upward and sends it to the positive side, but the label says negative.

The next two experiments are really the point of the lesson. Multiplying $\mathbf{w}$ and $b$ by ten multiplies every pre-activation by ten too ($2.0$ becomes $20$, $-1.0$ becomes $-10$), but it doesn't change a single decision: the boundary is the set where $z$ is zero, and which points are zero doesn't depend on the scale. Moving the bias to $-0.5$, on the other hand, slides the boundary half a point: the review sitting right on it falls to the other side, and the score climbs to ten out of ten.

The last few lines print the distance. With $\lVert \mathbf{w} \rVert = 2.449$, the first review lands $0.816$ from the boundary, while the movie is bad sits at $-0.408$: half as far, on the other side. That difference is right there in $z$, but none of it makes it through to $a$.

Test your intuition

Three questions: what a pre-activation is actually worth, what gets lost when you remove the bias, and what holds true about the boundary.

Two things are still left open, and they're not equally urgent. I wrote those eight weights above by eyeballing the reviews myself, and that trick won't work with a vocabulary of fifty thousand entries: getting a neuron to find its own weights from examples is what takes up most of the rest of this block. The other loose end is closer at hand. The geometry gave the magnitude of $z$ real meaning (the distance to the boundary, scaled by $\lVert \mathbf{w} \rVert$), and the step throws all of that away: it hands out the same $1$ to a review that falls at $0.816$ and to one that barely grazes the boundary.

$\varphi$ was fixed by decree, and there are plenty of other functions to choose from. Some preserve that distance instead of crushing it; others change, at the root, what becomes computable once you start stacking neurons together. Which function to assign to $\varphi$, and what each choice gains and loses, is the subject of the next lesson, on activation functions and non-linearity.