Why AI Chatbots Fabricate Sources, and How to Spot a Fake Citation

TL;DR: AI chatbots regularly fabricate citations: URLs, headlines, court cases, and academic papers that do not exist. The cause is structural. A language model generates the most plausible-sounding next words, and a citation is just more words. Without retrieval, the model has no real source to point to. The fix is not better prompting. It is wiring the model to a real search system and refusing to let it write source fields itself.

AI hallucinated sources are not a rare bug. They are the default output of any language model asked to back up a claim with a citation when it has no real source to point to.

In May 2023, a New York lawyer learned this the hard way. He filed a brief citing six federal court cases that looked entirely normal: named parties, plausible jurisdictions, neat citation format. None of them existed. He had used ChatGPT for research and trusted the output. The judge sanctioned him. The New York Times covered the moment when the lawyer realized the cases were not real.

That case is the most public example of a much wider phenomenon. Wherever a language model is asked to support a claim with a URL, a paper, or a case name, there is a meaningful chance the source is invented. The format is correct because the model has seen millions of real citations during training. The content behind the format is whatever the model finds most likely to come next.

This post explains why it happens, where it has caused real harm, how to spot a fake citation yourself, and what design choices keep modern fact-checking tools honest.

What Is an AI Hallucinated Source?

An AI hallucinated source is a citation produced by a language model where the format is correct but the article does not exist. The model has seen millions of real citations during training and knows the shape of one. What it does not know, without retrieval, is which citations actually point to real articles.

The surface looks identical to a real reference. A hallucinated court case has a case number that follows the right pattern. A hallucinated study has an author, a journal, a year, and a DOI-style identifier. A hallucinated URL has a sensible path on a real publisher's domain. Each piece is the kind of thing you would expect to see, which is exactly why fabrications slip past readers who do not click.

Researchers studying this phenomenon group it into categories. A widely cited survey of hallucination in natural language generation by Ji and colleagues describes intrinsic hallucinations (output that contradicts the input) and extrinsic hallucinations (output that introduces unverifiable content). Citations are the most damaging extrinsic case because they look like proof.

Why Do Chatbots Make Up Sources?

Language models generate the most plausible-sounding next words, one token at a time. A citation is just more words. Without a connection to a live search system, the model has nothing real to point to, so it produces something that fits the pattern of a citation rather than a verified one. The fabrication is structural, not malicious.

This is not a bug fixed by a sharper prompt or a stricter instruction. It is how the underlying architecture works.

A standard language model is trained to predict the next word in a sequence. When you ask it for a source, it produces tokens that look like a source. It has no internal database of what is real. It only has weights that encode statistical patterns from training data.

The well-known fix is retrieval-augmented generation, introduced by Lewis and colleagues in 2020. Their paper describes a system that retrieves real documents from an index, then generates an answer grounded in those documents. The architecture is now standard in serious AI products that need to cite sources. A worrying number of consumer chatbots still answer from training data alone.

The Real-World Damage

Hallucinated citations are not a theoretical concern. They have already produced sanctions, retractions, and bad rulings.

The Mata case is the headline example. The court fined the lawyer and his firm, and he apologised in open court. What unsettled observers most was not the technology failure. It was that the citations looked entirely normal to a trained legal professional until someone tried to read them.

A 2024 study from Stanford's Institute for Human-Centered AI looked at commercial legal research tools that advertise themselves as AI-powered. Even with retrieval, those tools hallucinated on a meaningful share of queries, with rates landing around one in six for the better-performing products. Stanford HAI publishes that work and other AI safety research at hai.stanford.edu.

Academic publishing has seen the same problem. Papers have been submitted with reference lists containing journals and authors that do not exist. Some were retracted. Some made it past review.

The pattern is consistent. Wherever the cost of a fake citation is high (court, peer review, medical advice), the damage shows up first. Wherever the cost is low (a casual chat with a search engine), the fakes mostly go unnoticed.

How to Spot a Fake AI Citation

To spot a fake AI citation, click the link. If it returns a 404, a wrong article, or a page that does not contain the cited content, you have a hallucinated source. Search the exact title in Google Scholar, the publisher's own site, or a court database. Real citations resolve; fabricated ones do not survive a single click.

The check is almost embarrassingly simple, and that is the point.

If the AI gave you a URL, open it. If the URL is dead, the source is fake. If the page loads but does not contain the cited claim, the source is fake.

If the AI gave you a paper title, search it. Google Scholar indexes most real academic work. If a title does not appear, that is a strong signal. If the title appears but the authors or year are different, the citation is mangled and untrustworthy.

If the AI gave you a court case, search the case name in a court database or a legal aggregator. Real cases leave a paper trail. Fabricated ones do not.

This is the same reflex as lateral reading: leave the source and check it from the outside. The faster you build the habit, the less time you waste on output that looks confident but cannot survive a verification step. For practical workflow tips on video claims, see our guide to verifying a YouTube claim.

What a Trustworthy AI Source Looks Like

A fact-checking tool that takes citations seriously is built differently from a chatbot.

The starting point is retrieval. Real articles get pulled from a live search before the model generates anything. The model is not asked to remember articles from training. It is given the actual text of articles found on the open web seconds earlier.

The second piece is source separation. The model is not allowed to produce the URL or publisher name directly. Those fields come from the search result. The model can only refer to retrieved sources by index. The server then resolves the index back to the real URL and publisher.

The third piece is honest failure. If no relevant sources come back, the tool refuses to issue a confident verdict. It says it cannot verify the claim rather than guessing.

That is the pattern behind the WasThatTrue Chrome extension. For a full walkthrough of the pipeline, see how the verdict pipeline retrieves sources. The design is not exotic, but it is deliberate, because every shortcut in this kind of system is where hallucinations creep back in.

Does Retrieval Solve the Problem?

Retrieval helps but does not eliminate the risk. A model with web search can still misquote a retrieved article, attach the wrong source to the wrong claim, or omit citations entirely. The architecture has to enforce that retrieval is the only source of URLs and the system has to refuse to answer when nothing relevant comes back.

The Stanford HAI study mentioned earlier looked specifically at retrieval-augmented legal AI products and still found significant hallucination rates. Retrieval is necessary; it is not sufficient.

The remaining failure modes are subtle. A model can paraphrase a retrieved article inaccurately. It can cite an article that exists but does not actually support the claim. It can confidently produce a verdict from one retrieved article while ignoring three others that contradict it. Each of these is harder to detect than a flat-out invented URL.

Open benchmarks track how often this happens. The Vectara hallucination leaderboard publishes comparative rates for leading models on summarisation tasks, where the input is a real document and the model's job is to stay faithful to it. Even on this constrained task, leading models still hallucinate on a measurable share of inputs.

The honest answer is that no current AI system is hallucination free. The useful design question is whether the tool fails loudly (refusing to answer, marking confidence low) or quietly (returning a confident wrong answer with a polished citation).

Frequently Asked Questions

Why do AI chatbots make up citations?

Language models generate the most plausible-sounding next words, one token at a time. A citation is just more words. Without a connection to a live search system, the model has no real source to point to, so it produces something that fits the pattern of a citation rather than a verified one. The fabrication is structural rather than a sign that the model is broken.

Is ChatGPT the only AI that hallucinates sources?

No. Hallucinated sources show up in every major commercial chatbot when they are asked to produce citations from training data alone. Public hallucination leaderboards track comparative rates across leading models, and none of them score zero. Hallucination is a property of the architecture, not a specific brand.

How common are AI hallucinated citations?

Common enough to cause repeat real-world incidents. A 2024 Stanford HAI study of commercial legal research tools, which already use retrieval, found that some products hallucinated on roughly one in six queries. Consumer chatbots without retrieval hallucinate more often. The rate depends on the task and the product.

Does retrieval-augmented generation fix the problem?

It reduces the risk but does not eliminate it. A model with live search can still misquote a retrieved article, attach the wrong source to the wrong claim, or ignore contradicting sources. The architecture has to do more than just retrieve: it has to refuse to answer when nothing relevant is found and stop the model from writing URL fields directly.

Can I trust an AI fact-checker if it shows me sources?

Only if you can click through. A tool that lists sources you cannot inspect is still a guess. A tool that shows you the actual article, publisher, and date for every claim it checks gives you the ability to verify the verdict yourself. That last step is what separates a fact-checker from a confident summariser.