The next-word game

In 4.1 we cut text into pieces. So what do we do with them?

Remarkably, only one thing. Guess the piece that comes next.

You can do the same thing. A storybook has been counted up and is waiting.

Not one rule anywhere

This machine knows no grammar. It doesn't know what any word means.

It only counted. And still, two or three words in, it sounds plausible.

A machine built to guess the next word is called a language model. Old ones and new ones do the same job.

Looking further back helps

Switch to 3 and build again. Much more natural, isn't it?

Of course — now it picks using all three previous words.

But read the note under the lab.

That's not good writing. That's copying

Those words are straight out of the storybook.

If those three words appear in the book exactly once, then only one word can follow. There's nothing to choose. It's just transcribing.

Counted up, it looks like this.

"Never appears"

So what happens if you try to build a sentence the book doesn't contain?

Add several words with 1 — that makes a sentence the book has never seen — then switch to 3.

It stops. About six times in ten it has nothing at all to say.

Reciting what it memorised, or giving up. One or the other.

It can't make a new sentence. That was the wall for counting.

So what's different now?

This time we'll put a small AI trained on that same storybook right beside it.

The same book matters. Otherwise you could say it only wins because it got better material.

The right side never gets stuck

Add any words of your own at the end. The left side hits "never appears" almost at once.

The right side answers whatever you give it. Even a sentence nobody has ever written.

That's because it isn't counting — it has adjusted weights. The same weights as in 3.2. It computes an answer instead of looking one up, so there are no dead ends.

And once the right side has written a line, the left side immediately says "never appears."

Naturally. The sentence the right side just made isn't in the book. The note under the lab says so too.

Counting at three words back copied the book 33% of the time. The right side: 0%.

Same sentence, different answer

Press Let it write on its own. Then Back to the start, and run it again.

Different, isn't it?

It doesn't always take first place — it draws by probability. Something at 70% gets picked about seven times in ten, and the rest of the time something else does.

Push How bold? to the top and it starts reaching for the lower choices. Then it goes strange.

Pull it all the way down and it always takes first place. Then it repeats itself over and over.

Make it good enough and…

Put like this it seems odd. What's so impressive about guessing the next word?

But make a machine very good at it, and something strange happens.

• Guess what follows "In English, this sentence reads:" → you get translation

• Guess what follows "In one line, that says:" → you get a summary

• Guess what follows someone's question → you get an answer

Nobody taught it to translate. It was only ever asked to guess the next word.

To see the whole sentence

Counting could only look three words back. Today's models look much further.

But a long sentence has dozens of words behind it. Does it weigh them all the same?

No. It chooses where to look. That's the next chapter.