How to tell when an AI answer is made up, in under two minutes
The bot sounds the same whether it is right or wrong. That is the whole problem. Here is the check I run before I trust anything that matters.
It makes things up and I can't tell. A five-step routine for checking an AI answer before you act on it: turn on search, ask for the exact quote, ask the bot to argue against itself, run the same question past a second model, and know which answers never to trust without a link.
Know which answers to doubt first
Not every answer needs checking. A recipe or a rewrite of your email is low risk. Anything with a number, a date, a name, a law, a price, a phone number or a quote is where the models fail, and they fail in a confident voice.
On October 26, 404 Media reported that ChatGPT had been telling people with banned Instagram accounts to email the magazine for help, with the magazine's real address. A journalist got the support tickets. In September the Financial Times tested chatbots on money questions and found they were wrong most of the time. Those are two of this month's examples. There will be more next month.
So the rule I use: if acting on the answer would cost money, time or embarrassment, run the rest of this check.
Turn search on and ask for the source
Every major assistant can now search the web, but the setting is not always on, and without it the model answers from memory. Memory is where hallucinations live.
In ChatGPT, press the + button in the message box and choose Search, or just write "search the web for this". In Claude, the Search toggle lives behind the + in the chat box too. In Gemini it is on by default for most questions. Perplexity searches by default and shows its sources as numbered links.
Then ask for the source in the same message. I use the exact words below, because "give me a source" gets you a plausible-looking link and "quote the sentence" gets you something you can check.
Search the web for this. For every fact in your answer, add the source URL and quote the exact sentence from that page that supports it. If you cannot find a page that says it, say "no source found" instead of guessing.
Open the link and find the sentence
This is the step people skip. A link in an answer is not evidence. The page has to exist, and it has to say what the bot says it says. In my testing the second part fails more often than the first: the page is real, the claim on it is not.
Click the link. Use your browser's find (Ctrl+F or Cmd+F) and paste a few words from the quoted sentence. If it is there, you are done with that fact. If it is not, the fact is unverified, whatever the bot's tone.
Make it argue against itself
Models agree with you. Ask "is this right?" and you will usually get "yes". Ask the question the other way round and you get the doubts the first answer hid.
Paste the answer back and ask it to attack it. If the attack finds nothing, that is weak evidence it holds. If the attack finds a hole, you have just saved yourself from it. This takes thirty seconds and catches about half of the mistakes I have seen.
Here is an answer you gave me earlier: [paste the answer] Assume it contains at least one error. List the three claims most likely to be wrong, say why each could be wrong, and tell me exactly what I would need to check to be sure.
Ask a second model, cold
Two models trained by two companies rarely make the same mistake in the same place. Open a different assistant, do not paste the first answer, and ask the original question again. Compare the parts that matter: the numbers, the names, the dates.
If they agree, trust goes up, not to certain, but up. If they disagree, one of them is wrong, and now you know exactly which fact to look up yourself. I keep Claude and ChatGPT open in two tabs for exactly this, and Perplexity for anything where I want the sources laid out first.
Get a confident answer.
Act on it.
Find out a week later it was wrong.
Open the link, Ctrl+F the sentence.
Ask the bot to attack its own answer.
Ask a second model cold and compare the numbers.
Now act on it.
TipIf a bot gives you a phone number or an email address to contact a company, never use it without finding it on the company's own site. Scam pages are now written to be picked up by AI search, and this is how they get you.
You're done when
Questions people are asking
Why does ChatGPT make things up? A language model predicts likely text. When it does not know, the most likely text is still a confident sentence, not 'I don't know'. Search grounding and asking for quoted sources reduce this but do not remove it.
Does turning on search stop hallucinations? It helps a lot for facts that exist on the web, because the model can quote instead of guess. It does not help when the web page itself is wrong or planted, which is why you still open the link.
Which AI is the most accurate? In our use, none is reliable enough to skip the check for anything that matters. Perplexity shows sources best. Claude and ChatGPT with search on are close. The routine above matters more than the brand.
Sources
- 404 Media, ChatGPT keeps telling people 404 Media can solve their tech support problems, Oct 26 2026 404media.co/chatgpt-keeps-telling-people-404-media-can-solve-their-tech-support-problems
- Financial Times, chatbots and financial questions, Sep 21 2026 ft.com/content/c0cd359d-df84-4208-a789-ffa864b43666
- Rev, survey on how often people double-check AI results rev.com/blog/ai-results
How to make ChatGPT, Claude and Gemini remember you (and forget what you want)
One email a week. Written by people.
New how-tos, the tools we tested and what we think, in one short email.