On 8 October 2026, an audit of this site turned up 76 wrong claims about our own prompts across 31 pages. A profession page would send readers to a prompt for a job it doesn't do, or promise five or six headlines from a prompt that writes ten. I'd drafted those pages with Claude Code, an AI coding agent, and the audit ran on AI too: four read-only subagents split the site between them, and each finding was re-checked against the prompt it named.
That audit changed how I fact-check AI-written content about our own products. Since then I've fixed the pages one at a time, 18 so far, checking each sentence against the prompt it describes.
What counts as a wrong claim about a prompt?
A wrong claim, in this audit, is a sentence on one page that says one of our prompts does something that prompt's own page shows it doesn't. Every prompt page carries the prompt itself, a short section on what it does, and a list of variations, so there's always something to check against. If a variation covers the claim, it isn't wrong.
Six candidates were cut on re-check before the audit was written up. A freelance-writers FAQ that used the follow-up prompt on past clients was one, because the prompt has a "Re-engage a cold lead" variation. A line telling entrepreneurs to "ask for three lengths" was another: it's an instruction to the reader, not a claim about the prompt.
Three that survived, all fixed since:
| Page | What the page said | What the prompt does |
|---|---|---|
| Real estate agents | The headline variations prompt would give "five or six ways" to title a listing | Writes 10 headlines, two in each of five formulas |
| Virtual assistants | The weekly review prompt "turns your own scrappy log into the client-facing summary" | Reviews your own week (done, not done, what derailed it), then names a pattern and plans the next one |
| Teachers | The follow-up email prompt was the one to write up a parent conference's agreed plan | Nudges someone who hasn't replied: under 120 words, value first |
The teachers page now sends the conference plan to the meeting summary prompt, which lists the decisions and gives each action item a named owner and a due date.
Why did the claims drift?
Most of the wrong claims were guesses that a prompt's name makes easy.
Six prompts account for 34 of the 76, and two kinds of guess cover all six:
| Prompt | Wrong claims | What pages said | What it does |
|---|---|---|---|
| Follow-up email | 10 | Meeting recaps, status updates | A nudge after no reply |
| Weekly review | 9 | Pipeline reviews, client summaries, ticket themes | Your own week |
| FAQ generator | 5 | Employee policy questions, student quizzes | A product or service FAQ (pricing, refunds, delivery) |
| Onboarding checklist | 3 | Client intake, setting up a new process | A new employee's first months |
| Headline variations | 4 | Five or six, a dozen, 8 to 10, twenty | Ten |
| Explain a concept | 3 | Three depths, two or three analogies | One explanation with one analogy |
The first four got a job their name suggests. "Follow-up email" sounds like any email that follows anything, and "weekly review" sounds like whatever your team reviews on Fridays. The last two got a number wrong, and the numbers they got were round, plausible and never the one in the prompt.
That fits the explanation in Why Language Models Hallucinate, a September 2025 paper by Adam Tauman Kalai and three co-authors. Models "sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty", they argue, because training and evaluation reward the guess. A prompt called "follow-up email" that writes your meeting follow-up is about as plausible as a guess gets.
Did the audit miss any?
An audit's miss is a wrong claim on a page it read that never made its list, and on the 18 pages fixed so far it missed 6. Getting to that number took a recount.
Each fix pass logged the problems it found that weren't on the audit's list, and those logs added up to 30 sentences. For this post I re-read all 30 against the audit's own rule. Six held. On the designers page, the elevator pitch prompt was said to tighten "the thirty-second version of a concept you'll repeat at every review", but it pitches you: your work, the people you serve, the problem you solve. The brainstorming prompt, on the real estate page, was said to help you "anticipate what they'll push on", when it runs three rounds of ideas and never raises an objection. Four were smaller slips: two prompts with only a tone setting credited with holding a brand's or a client's voice, the weekly review sold as raw input for client reports, and a repurposing prompt that writes three video hooks credited with "a reel".
The other 24 didn't hold up, for four reasons.
| Why it didn't count | Sentences | One example |
|---|---|---|
| The prompt's own page covers it | 9 | The grammar check prompt called "the fast pass": its Quick mode variation skips the explanations |
| Accurate, or close enough | 10 | Business proposal: "structures responses to RFPs", and its own page names RFP responses among its uses |
| Wrong the other way | 2 | The task prioritization prompt "won't make the call for you", when it ends with a suggested order |
| An instruction to the reader | 3 | "Ask the explain prompt for two extension questions" |
The nine in the first row share a cause. The fix passes checked each claim against the prompt box and stopped there, so the variation or note further down the page that made the claim true never got read. The audit's rule had been to read the variations too. The same habit took a true line about RFPs off the accountants page. The two in the "wrong the other way" row are real errors, just not the kind the audit was looking for, so they stay out of its count.
It isn't the first count from an AI agent on my sites to shrink on a recount. In June an agent counted 140 duplicate lines, and a recount found zero. Here 30 logged misses came down to 6.
How do you fact-check AI-written content about your own product?
Fact-checking AI-written content about your own product means finding, for every claim, the line in the product's own documentation that supports it.
- Pull out every sentence that names a product, and split it into its separate claims. "Drafts posts in the client's voice rather than the default LinkedIn register" makes two. FActScore, a 2023 method for scoring long AI-written text, works the same way: it breaks a passage into atomic facts and checks each one against a source, because a single passage usually mixes supported and unsupported pieces.
- Open the product's own page and find the line behind each claim, in the variations and usage notes as well as the main description. No line, no claim. Anthropic's guide to reducing hallucinations gives the same instruction for a product press release: find a direct quote for each claim, and remove any claim that doesn't have one.
- Fix it one of three ways. Say what the prompt actually does, send the job to the prompt that does it, or take the prompt off the page. Meeting recaps moved from the follow-up prompt to the meeting summary prompt, and the FAQ generator came off the teachers page and two others because nothing honest was left for it there.
- Write down the rule for what counts before anyone counts, and hold your own fixes to it. Our fix passes didn't, and that's where 24 of the 30 came from.
Claims about the outside world, like a statistic or a quote someone gave, need a different check, and the research guide's verification pass is where that one lives.
Follow-up email has the plainest name on the shelf and drew more wrong claims than any other prompt, 10 of the 76. Its second bullet reads: "It's been [TIME SINCE LAST CONTACT …] with no reply."
Sources
- Anthropic: Reduce hallucinations, Claude documentation, the "Verify with citations" technique (read 13 November 2026)
- Min et al.: FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, arXiv, May 2023
- Kalai, Nachum, Vempala and Zhang: Why Language Models Hallucinate, arXiv, September 2025
- The audit (8 October 2026), the 18 fix-pass task files and the recount are kept in this site's repository.
Counted 13 November 2026 from the audit file and the 18 fix-pass task files, with each of the 30 logged misses re-read against its prompt's own page.
Common questions
- How do you fact-check AI-written content about your own product?
- Find the line in the product's own documentation that supports each claim, and cut or rewrite any claim that has no such line. Split long sentences into their separate claims first, and read the variations and usage notes as well as the main description, or you'll flag claims that are actually true.
- Why does AI get descriptions of your own products wrong?
- It writes the plausible version, and a product's name makes the plausible version easy to reach for. On this site, six prompts accounted for 34 of 76 wrong claims, most of them jobs the prompt's name suggests, like a follow-up email prompt described as writing meeting recaps when it only nudges people who haven't replied.
- Can an AI audit catch every wrong claim on a site?
- Not every one. An audit of 31 pages here listed 76 wrong claims, and re-reading 18 of those pages over the following five weeks found 6 more under the same rules. The recount mattered too: 30 sentences had been logged as misses, and 24 of them didn't hold up.