TameTheBot
Blog

Our AI-Written Pages Got Our Own Prompts Wrong

An AI audit found 76 wrong claims our AI-drafted pages made about our own prompts. Why they drifted, and the sentence-by-sentence check we use now.

8 min readPublished November 13, 2026

On 8 October 2026, an audit of this site turned up 76 wrong claims about our own prompts across 31 pages. A profession page would send readers to a prompt for a job it doesn't do, or promise five or six headlines from a prompt that writes ten. I'd drafted those pages with Claude Code, an AI coding agent, and the audit ran on AI too: four read-only subagents split the site between them, and each finding was re-checked against the prompt it named.

That audit changed how I fact-check AI-written content about our own products. Since then I've fixed the pages one at a time, 18 so far, checking each sentence against the prompt it describes.

What counts as a wrong claim about a prompt?

A wrong claim, in this audit, is a sentence on one page that says one of our prompts does something that prompt's own page shows it doesn't. Every prompt page carries the prompt itself, a short section on what it does, and a list of variations, so there's always something to check against. If a variation covers the claim, it isn't wrong.

Six candidates were cut on re-check before the audit was written up. A freelance-writers FAQ that used the follow-up prompt on past clients was one, because the prompt has a "Re-engage a cold lead" variation. A line telling entrepreneurs to "ask for three lengths" was another: it's an instruction to the reader, not a claim about the prompt.

Three that survived, all fixed since:

PageWhat the page saidWhat the prompt does
Real estate agentsThe headline variations prompt would give "five or six ways" to title a listingWrites 10 headlines, two in each of five formulas
Virtual assistantsThe weekly review prompt "turns your own scrappy log into the client-facing summary"Reviews your own week (done, not done, what derailed it), then names a pattern and plans the next one
TeachersThe follow-up email prompt was the one to write up a parent conference's agreed planNudges someone who hasn't replied: under 120 words, value first

The teachers page now sends the conference plan to the meeting summary prompt, which lists the decisions and gives each action item a named owner and a due date.

Why did the claims drift?

Most of the wrong claims were guesses that a prompt's name makes easy.

Six prompts account for 34 of the 76, and two kinds of guess cover all six:

PromptWrong claimsWhat pages saidWhat it does
Follow-up email10Meeting recaps, status updatesA nudge after no reply
Weekly review9Pipeline reviews, client summaries, ticket themesYour own week
FAQ generator5Employee policy questions, student quizzesA product or service FAQ (pricing, refunds, delivery)
Onboarding checklist3Client intake, setting up a new processA new employee's first months
Headline variations4Five or six, a dozen, 8 to 10, twentyTen
Explain a concept3Three depths, two or three analogiesOne explanation with one analogy

The first four got a job their name suggests. "Follow-up email" sounds like any email that follows anything, and "weekly review" sounds like whatever your team reviews on Fridays. The last two got a number wrong, and the numbers they got were round, plausible and never the one in the prompt.

That fits the explanation in Why Language Models Hallucinate, a September 2025 paper by Adam Tauman Kalai and three co-authors. Models "sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty", they argue, because training and evaluation reward the guess. A prompt called "follow-up email" that writes your meeting follow-up is about as plausible as a guess gets.

Did the audit miss any?

An audit's miss is a wrong claim on a page it read that never made its list, and on the 18 pages fixed so far it missed 6. Getting to that number took a recount.

Each fix pass logged the problems it found that weren't on the audit's list, and those logs added up to 30 sentences. For this post I re-read all 30 against the audit's own rule. Six held. On the designers page, the elevator pitch prompt was said to tighten "the thirty-second version of a concept you'll repeat at every review", but it pitches you: your work, the people you serve, the problem you solve. The brainstorming prompt, on the real estate page, was said to help you "anticipate what they'll push on", when it runs three rounds of ideas and never raises an objection. Four were smaller slips: two prompts with only a tone setting credited with holding a brand's or a client's voice, the weekly review sold as raw input for client reports, and a repurposing prompt that writes three video hooks credited with "a reel".

The other 24 didn't hold up, for four reasons.

Why it didn't countSentencesOne example
The prompt's own page covers it9The grammar check prompt called "the fast pass": its Quick mode variation skips the explanations
Accurate, or close enough10Business proposal: "structures responses to RFPs", and its own page names RFP responses among its uses
Wrong the other way2The task prioritization prompt "won't make the call for you", when it ends with a suggested order
An instruction to the reader3"Ask the explain prompt for two extension questions"

The nine in the first row share a cause. The fix passes checked each claim against the prompt box and stopped there, so the variation or note further down the page that made the claim true never got read. The audit's rule had been to read the variations too. The same habit took a true line about RFPs off the accountants page. The two in the "wrong the other way" row are real errors, just not the kind the audit was looking for, so they stay out of its count.

It isn't the first count from an AI agent on my sites to shrink on a recount. In June an agent counted 140 duplicate lines, and a recount found zero. Here 30 logged misses came down to 6.

How do you fact-check AI-written content about your own product?

Fact-checking AI-written content about your own product means finding, for every claim, the line in the product's own documentation that supports it.

  1. Pull out every sentence that names a product, and split it into its separate claims. "Drafts posts in the client's voice rather than the default LinkedIn register" makes two. FActScore, a 2023 method for scoring long AI-written text, works the same way: it breaks a passage into atomic facts and checks each one against a source, because a single passage usually mixes supported and unsupported pieces.
  2. Open the product's own page and find the line behind each claim, in the variations and usage notes as well as the main description. No line, no claim. Anthropic's guide to reducing hallucinations gives the same instruction for a product press release: find a direct quote for each claim, and remove any claim that doesn't have one.
  3. Fix it one of three ways. Say what the prompt actually does, send the job to the prompt that does it, or take the prompt off the page. Meeting recaps moved from the follow-up prompt to the meeting summary prompt, and the FAQ generator came off the teachers page and two others because nothing honest was left for it there.
  4. Write down the rule for what counts before anyone counts, and hold your own fixes to it. Our fix passes didn't, and that's where 24 of the 30 came from.

Claims about the outside world, like a statistic or a quote someone gave, need a different check, and the research guide's verification pass is where that one lives.

Follow-up email has the plainest name on the shelf and drew more wrong claims than any other prompt, 10 of the 76. Its second bullet reads: "It's been [TIME SINCE LAST CONTACT …] with no reply."

Sources

Counted 13 November 2026 from the audit file and the 18 fix-pass task files, with each of the 30 logged misses re-read against its prompt's own page.

Common questions

How do you fact-check AI-written content about your own product?
Find the line in the product's own documentation that supports each claim, and cut or rewrite any claim that has no such line. Split long sentences into their separate claims first, and read the variations and usage notes as well as the main description, or you'll flag claims that are actually true.
Why does AI get descriptions of your own products wrong?
It writes the plausible version, and a product's name makes the plausible version easy to reach for. On this site, six prompts accounted for 34 of 76 wrong claims, most of them jobs the prompt's name suggests, like a follow-up email prompt described as writing meeting recaps when it only nudges people who haven't replied.
Can an AI audit catch every wrong claim on a site?
Not every one. An audit of 31 pages here listed 76 wrong claims, and re-reading 18 of those pages over the following five weeks found 6 more under the same rules. The recount mattered too: 30 sentences had been logged as misses, and 24 of them didn't hold up.
Rakesh Kumar Maity
Rakesh Kumar Maity

Founder and editor of TameTheBot, and a software developer in Kolkata. LinkedIn

Found this useful?

Prompts from this post

Related guides