What an AI hallucination actually is
A hallucination is output that reads as confident, specific, and well-formed while being factually wrong or entirely invented. The term covers a wide range: a misremembered date, a statistic with no origin, a quote attributed to someone who never said it, a study that doesn't exist, a court case with a real-sounding name and a fabricated docket number. Researchers have been cataloguing the problem across text generation tasks for years, including in the widely cited Survey of Hallucination in Natural Language Generation.
What makes it dangerous for research work isn't the error rate. It's that the errors arrive wearing the same clothes as the correct answers. A wrong statistic is formatted exactly like a right one. There's no hedge in the tone, no visible seam.
$5,000 is what it cost two New York lawyers in June 2023, when a federal judge sanctioned them for submitting a brief built on judicial opinions that ChatGPT had invented, complete with fabricated quotes and citations. The sanctions opinion in Mata v. Avianca is worth reading in full, because the failure wasn't really the model's. Asked to confirm the cases were real, the lawyer asked ChatGPT, which said yes.
That's the pattern this guide is built around. The model produced a plausible answer, the human verified it with the same model, and the loop closed on itself with nothing outside it ever being checked.
Why the model invents a citation instead of saying "I don't know"
A language model generates the next piece of text by predicting what's statistically likely to follow, which means a fabricated citation and a real one come out of an identical process. It isn't retrieving a record from a database and it isn't consulting an index. It's producing the most plausible continuation of your prompt, and a well-formed citation is an extremely predictable continuation of a request for sources.
Two consequences follow, and both are useful to hold onto.
The first: absence of knowledge doesn't feel like anything to the model. When it has seen a fact many times, and when it has seen nothing at all, the generation process is the same. Silence isn't a natural output. Something plausible is.
The second: fluency and accuracy are separate properties that happen to be produced by one system. Fluency is what the training objective optimizes directly. Accuracy comes along for the ride when the training data was accurate and the question is common. On a question that's rare, recent, or oddly specific, the ride ends and the fluency keeps going.
This is also why the model vendors ship grounding and retrieval features rather than simply training the problem away. Anthropic's own guidance on reducing hallucinations leads with giving Claude permission to say it doesn't know and asking for direct quotes from supplied documents. Google's Gemini grounding documentation and OpenAI's retrieval guide describe the same underlying move: connect the answer to a real document or a live search result instead of the weights.
You can apply that move without any API access. It's mostly a matter of what you paste and what you ask for.
The five techniques that cut fabrication most
1. Hand it the source instead of asking it to remember
The single highest-return change in AI research work: stop asking questions that require the model to recall facts, and start asking questions about text you've supplied. "What does the attached report say about churn in Q3" is a fundamentally safer question than "what's the average SaaS churn rate," because the first can be checked against something in front of you and the second can't be checked at all.
Practically, that means finding the source first (your library database, the actual PDF, the company's own filing) and pasting it in. The model becomes a reading assistant rather than an oracle. Our prompt for summarizing a long document is built for exactly this pattern, and the book summary prompt works the same way when you're working from a text you have.
2. Make "I don't know" a permitted answer
Models default to answering. Left to itself, a model treats an unanswerable question as a hard question rather than an impossible one, and generates accordingly. One explicit instruction changes the payoff:
If any part of this question can't be answered from the text I provided, say "not in the source" for that part instead of filling it in. Do not use outside knowledge.
Adding a permitted escape hatch costs you nothing when the answer is present and saves you the fabrication when it isn't. Anthropic's documentation recommends the same instruction for the same reason.
3. Ask for the quote, not the claim
Requesting a claim invites a summary, and a summary is where a small drift becomes a wrong fact. Requesting the underlying quote forces the model to point at specific text you can find with a search on the page:
For each point you make, include the exact sentence from the source that supports it, in quotation marks, with the section heading it appeared under. If you can't find a supporting sentence, drop the point.
Quotes have a property claims don't: they're checkable in about four seconds. Copy the sentence, hit find-in-page on the original, and you either land on it or you don't.
4. Work one source at a time
Give a model six papers and a broad synthesis question and it will blend them, which is where invented cross-references come from: a finding from paper two attributed to paper five, a methodology that belongs to neither. Process each source separately, get a short structured summary of each with quotes attached, then do the synthesis yourself (or in a final pass where all six summaries are pasted in as the only permitted input).
It's slower per source and much faster overall, because you never have to unpick a blended claim later.
5. Verify in a different model, never the same one
Asking the model that produced a claim to check that claim is the loop that cost the Mata lawyers $5,000. The check runs on the same weights that generated the error, so a confident mistake gets confidently confirmed. Pasting the claim into a second tool at least gives you an independent generation, and disagreement between two models is a genuinely useful signal that a claim needs a human look.
Independent generation still isn't verification. Two models can be wrong in the same direction, especially on something that was widely repeated in training data. Treat cross-model agreement as a reason to keep going, and treat the original source as the only thing that settles it.
What this looks like on a real question
Take a concrete task: you're writing a literature review and you need to know what a specific 2023 paper concluded about factual precision in long-form model output.
Asked from memory, the request produces a fluent paragraph about the paper's findings, possibly with a percentage attached, possibly citing authors in the right field. Whether any of it corresponds to the paper is unknowable from the output alone.
The grounded version takes three extra minutes. Search for the paper, land on FActScore on arXiv, download the PDF, and paste it in with a bounded question:
Here's the full text of a paper. Answer only from this text. (1) What problem do the authors say they're solving? (2) What method do they propose? (3) What are their main reported results, with the exact sentence for each, in quotation marks? If something isn't stated in the text, write "not stated."
What comes back is checkable line by line, and the "not stated" entries tell you where your own reading still has to happen. The difference between the two versions isn't the quality of the prose. Both read well. The difference is that one of them has a paper trail.
The verification pass: four checks before you use anything
Run these on any AI output that's going into work with your name on it.
Does the source exist?
Open it. Not the summary of it, the thing itself. A DOI that resolves, a URL that loads, a case that appears in the docket. Fabricated citations fail this check immediately, which is why it goes first.
Does the source say what the output claims?
Existence isn't agreement. A real paper can be cited for a finding it never reported, and this failure is harder to catch than an invented citation because the reference checks out at a glance.
Is the number attached to a date and a source?
Any statistic without both is unusable. If the model produced a figure and can't tell you where it came from when asked directly, drop the figure rather than hunting for a source that supports it, which is a good way to talk yourself into a wrong number.
Would you stake your reputation on it?
The Mata brief passed every internal test the lawyers applied. It failed the only one that mattered, which was whether anyone outside the loop had looked.
Where AI research fails hardest
Failure isn't uniform across question types. These are the categories where the fabrication rate climbs and the checking has to be strictest.
| Question type | What goes wrong | What to do instead |
|---|---|---|
| "Find me sources on X" | Plausible citations for papers that don't exist | Search a real database, then paste what you find |
| Specific statistics | Numbers with no origin, or real numbers with wrong dates | Go to the primary publisher, cite the year |
| Recent events | Training cutoffs mean confident answers about a world that moved on | Use a search-connected tool, then open the links |
| Direct quotes | Paraphrases presented as verbatim, attributed to the wrong person | Find the quote in the original text yourself |
| Niche technical detail | Version numbers, parameters, and API details drift | Check the vendor's current documentation |
| Legal, medical, financial specifics | Confident answers in areas where being wrong causes real harm | Ask a qualified professional; this site doesn't advise on these |
That last row is a hard line rather than a technique. No prompt in this guide makes an AI tool a safe source of legal, medical, or financial guidance, and the failure mode there isn't an embarrassing citation, it's a decision that hurts someone.
Research prompts you can copy
Three prompts covering the common shapes of research work. Each assumes you've already got the source in hand.
To extract structured findings from one document:
Below is a source document. Answer only from this text, and write "not stated" for anything it doesn't cover. Give me: (1) the central claim in one sentence, (2) the evidence offered for it, with the exact supporting sentence in quotation marks for each piece, (3) any limitation the authors acknowledge, (4) what a skeptical reader would push back on. Document: [paste]
To pressure-test your own argument:
Here's an argument I'm making in a paper. Give me the three strongest objections a well-informed critic would raise, and for each one, tell me what kind of evidence would actually answer it. Don't invent sources or cite studies. Argument: [paste]
To turn a dense source into something you can study from:
Convert the text below into study material: key terms with plain-English definitions, the relationships between the main concepts, and five questions that would test real understanding rather than recall. Use only the text provided. Text: [paste]
The second one is deliberately source-free, and the "don't invent sources" instruction matters there: the moment you ask for objections, the model's next instinct is to attach citations to them. Our explain a concept prompt and study guide prompt cover the same ground in finished form, and if you're new to structuring prompts at all, the beginners guide covers the variables these prompts are using.
Coursework and faculty work pull on different parts of this. The prompts for students hub leans toward the reading, study, and drafting side, while the prompts for college professors hub covers the research writing, grant, and recommendation work that sits around it.
Picking a tool for research work
Tool choice matters less than mode, but it isn't nothing. A model with a large context window handles a full paper in one paste. A search-connected tool gives you links you can open. A comparison of how ChatGPT, Claude, and Gemini handle research and summarization specifically is in our tool comparison guide, which goes through the tradeoffs task by task.
Whichever you land on, the setup does the work: source pasted in, "I don't know" permitted, quotes requested, citations opened.
Action steps
- Take the last piece of AI output you used in real work and run the four verification checks on it. If a source fails check one, that's your baseline.
- Save the "not in the source" instruction somewhere you can paste it. Adding it to a prompt takes five seconds.
- On your next research question, find the source first and paste it in, rather than asking the question cold.
- Pick a second tool as your cross-check, and never use the tool that produced a claim to verify that claim.
- If you're citing AI output in academic work, read the APA guidance on citing ChatGPT and check your institution's own policy.
The tell that you've internalized this: you stop asking AI tools what's true and start asking them what a document you already have in front of you says.
Sources
- Anthropic, Reduce hallucinations: official guidance on allowing "I don't know" and requesting direct quotes.
- Google AI, Grounding with Google Search: Gemini documentation on connecting answers to live search results.
- OpenAI, Retrieval: documentation on answering from supplied documents rather than model memory.
- APA Style, How to cite ChatGPT: the citation format for large language model output.
- United States District Court, Southern District of New York, Mata v. Avianca, Opinion and Order on Sanctions, 22-cv-1461 (PKC), filed 22 June 2023: the $5,000 sanction for fabricated citations.
- Ji et al., Survey of Hallucination in Natural Language Generation: academic survey of the problem across text generation tasks.
- Min et al., FActScore: the paper used as the worked example above.
