Inside a neural network

In 3.1 you set 3,075 weights. But you never saw where those weights were attached.

Let's open it up. It's far simpler than it sounds.

One little calculator

Take in a few numbers, multiply each by a weight, add them all up, send out one answer. There's a little calculator that does exactly that and nothing else.

One of those calculators is called a neuron.

Make a lot of them and wire them together in layers, and that's a neural network.

The weights sit on the wires — one per connection. Those 3,075 weights in 3.1 were 3,075 connections.

The line moved by itself until it found its place.

Look at the picture on the right. Some lines are thick, some are thin. That thickness is the weight. Thicker means it got turned further.

That's the same thing you turned by hand in 1.2, sitting right there. This lab has two of them; 3.1 had 3,075; a real AI has millions.

One neuron draws one straight line

Here's the thing that matters. One neuron can only ever draw one straight line.

Multiplying and adding is all it does, and that arithmetic can't draw anything more complicated than a straight line.

So far that's been fine. Red on the left, blue on the right — one straight line splits them.

But what about an arrangement like this?

Press Example · diagonal and train it again. The board is still up there.

The line keeps wobbling. It never settles. Hundreds of rounds and it still can't.

One straight line cannot split this arrangement. Try drawing one yourself. However you draw it, one side comes out mixed.

What happened in 1969

That 1958 machine has a name: the perceptron. One neuron — the simplest neural network there is.

In 1969 two researchers proved exactly what you just ran into. A perceptron can only draw one straight line, so it cannot solve this, one of the easiest problems there is.

The bigger the hopes had been, the bigger the disappointment. Money and people left, and neural network research froze for close to twenty years.

One button

Now let's solve it.

It works.

The boundary isn't a straight line any more. It curves. Adding one layer of neurons let the machine combine several straight lines.

That button you just pressed took the world twenty years to press.

Back to 2.2

At the end of 2.2 we parked a question.

Who picks the numbers inside a filter?

Now we can answer it. Nobody picks them. Training picks them.

Those nine numbers in a filter are weights too. They start out as anything at all, and every mistake nudges them a little. Do that long enough and one filter ends up finding vertical lines, another finds horizontal ones. Nobody told them to.

Stack filters like that in layers and the network has a name: a CNN. That's what you were opening up, layer by layer, back in 2.2.

So just add loads of layers?

Adding a layer solved it. So would piling on layers and neurons solve anything at all?

It doesn't work like that. Next chapter, what happens when there are too many weights.