Confidently wrong

Have a chat with the counsellor first. Explanations later.

This counsellor is from 1966

Before your parents were born.

Press Open the rule list. That's all of it.

Around twenty rules. "If there's a word for a family member → Tell me more about your family." Twenty-odd lines like that, and nothing else.

Each reply now shows which rule fired. Scroll back through your conversation.

It didn't understand one word you said.

If it makes sense, we assume a mind

When we hear a reply that makes sense, we assume there's something behind it doing the thinking.

That's one of the better things about people. Here it's the reason they were fooled.

So what about today's chatbots?

It explains a book that doesn't exist

What kind of book is The Library on the Twelfth Stair?

The plot, the year, the award it won — all of it, without hesitating.

Now press Where did you read that?

It invents the evidence too

A journal name, and which volume.

That's far more frightening than getting the facts wrong. It reassures the person who tried to check with evidence that isn't there.

Press Check whether it's true. No such book, no such award, no such journal.

Stating something untrue as if it were fact is called hallucination.

But one answer was right

It got the boiling point of water right.

That matters. AI isn't wrong all the time. Most of the time it's right.

If it were always wrong this would be easy — you'd just never believe it. It's hard because it's only sometimes wrong.

So which is more dangerous?

Which is why you check

There's only one method. Check anything that matters somewhere else.

These especially:

• Numbers — dates, counts, arithmetic

• Names — of people, books, places

• Sources — the claim about where it read it

Make one yourself

Last thing — invent a book that doesn't exist.

Give it a title, write a plausible plot, attach a plausible source.

Not hard, is it. What a chatbot does is exactly what you just did. Once you've built one, you can spot one.

Would asking better help?

Same AI, but the answer shifts quite a lot depending on how you ask.

That's the next chapter.