Check an AI answer for made-up facts in 15 minutes
https://patrickz.ai/learn/understand-how-ai-answers/

OUTCOME
See why a fluent answer can still be wrong, then check an AI’s claims outside the chat.

YOU NEED
Any chat assistant, a web browser for checking, and a topic you know well. No coding required.

1. Observe a continuation
Ask the assistant to continue “The bank was…” in a river scene and then in a finance scene. Compare the words that become plausible when context changes.

2. Ask for checkable facts
Ask: “Recommend three books about [a hobby or topic you know well], with exact title, author and year.” Save the reply exactly as given.

3. Check every item outside the chat
Look up each title and author in a library catalog or bookstore search, not in the same chat. Mark each one found as described, found with different details, or not found. A tidy, confident list is not evidence.

4. Add a rule and compare
In a new chat, repeat the request with: “Only list books you are confident exist; write not sure instead of guessing.” Check the new list the same way. The rule can reduce errors; it cannot replace your check.

PROMPT
Recommend three books about [topic you know well]. For each give the exact title, author and year of publication.

TRY IT NOW (READY-MADE SAMPLE)
Recommend three books about growing tomatoes on a balcony. For each give the exact title, author and year of publication.

EXAMPLE / EXPERIMENT
A reply can pair a real author with a book they never wrote, in perfect formatting. If any item is “not found” or has the wrong year, you have seen the lesson: the model predicts plausible text, and checking it is your job. If all three check out, your check is still what made them trustworthy; try a narrower topic next time.

A GOOD ANSWER INCLUDES
- Each book marked found, different or not found, based on a catalog search
- One sentence on what the “not sure” rule changed

CHECK YOUR RESULT
[ ] Every listed book was looked up outside the chat and marked found, different or not found.
[ ] You can say what the “not sure” rule changed and what it did not.
[ ] You can explain how context changed the continuation.

IF IT FAILS
If the assistant searches the web and shows links, open each one and confirm it describes that exact book; a link is not a check until you read it. If you cannot confirm a book exists, mark it unverified instead of trusting the chat.

MY RESULT / NEXT CHANGE