The bot states the return window confidently. You open the citation and find a blog post that mentioned returns in passing — not the policy page your team maintains. Support closes the ticket; the visitor got the right number for the wrong reason. Next week someone paraphrases differently and the answer changes.
Wrong-page retrieval is the most common “almost working” failure we see. It is rarely fixed by prompt engineering because the model did its job: synthesise from the chunk that won the search. The bug is upstream — what you indexed, how you chunked it, and which pages compete for the same words.
How the wrong page wins
Embeddings and keyword retrieval both favour pages that repeat the visitor’s language, even when they do not state the fact. A marketing page titled “Easy Returns” that never lists the window beats a dry policy page if the policy uses different vocabulary. Tag archives, related-article modules, and footers that repeat product names add noise that looks like signal.
- Duplicate topics across FAQ, blog, and product copy.
- Long pages where the relevant sentence is buried under SEO paragraphs.
- Old pages still indexed after a rewrite.
- PDFs and slide decks with friendly wording but no authority.
Diagnose with citations, not vibes
When someone reports a bad answer, ask one question first: which URL did it cite? If the URL is wrong, stop tuning tone. Open the page and highlight the sentence the bot used. If the sentence is not there — only a thematic cousin — you have a chunking or hallucination boundary case. If the sentence is there but outdated, you have a recrawl problem.
Fix the index before the prompt
Remove or exclude pages that should not be authorities: campaign landings, tag pages, internal wikis, staging hosts. Promote the canonical policy URL in your crawl scope — what to index first is the prioritisation version of this fix.
Merge or redirect duplicates so one page owns the fact. Edit losers to say “see canonical policy” without restating numbers — repetition is how losers keep winning retrieval. After structural changes, recrawl and rerun your paraphrase set.
Chunking and page structure
Pages that mix twelve topics in one URL produce chunks where half a paragraph is shipping and half is returns. The model answers from whichever half retrieved. Split pages by intent, one question per heading hierarchy, so chunks align with questions. Chunking mistakes covers the failure modes that look like model stupidity.
Put the definitive sentence near the heading visitors ask about. “How long do I have to return?” should face a line that states the window, not three screens of brand story.
When to boost, block, or pin
Some products let you pin or boost URLs for topics. Use sparingly: pins fix emergencies, not architecture. Blocking bad URLs is healthier than permanently boosting good ones — otherwise every launch adds another override layer nobody documents.
If you need more than a handful of pins, your content map is fighting the bot. Invest in docs structure instead of chasing each miss on its own.
Close the loop with support
Tag tickets where the citation URL was wrong, not just where the answer text was wrong. Monthly, rank those URLs by frequency and fix or exclude them. Support already knows which pages lie; retrieval just made the lie visible.
When marketing launches a campaign page that reuses product vocabulary, add it to the watch list before crawl picks it up organically. Preemptive exclusion is cheaper than post-hoc apology emails.
“A right answer from the wrong page is a countdown to a public wrong answer.”
Playground habits that scale
Keep a saved list of ten paraphrases per high-risk topic — returns, warranty, compatibility. Run it after every crawl that touches those URLs. Thirty seconds per question beats a public complaint. Share results in Slack or email so the whole team sees citation drift, not just whoever owns the bot.
SEO pages versus help pages
SEO content repeats keywords; help content states facts. When both exist for the same topic, search engines may prefer either — your bot will too. Canonicalize in content strategy: SEO pages link to help; help pages do not rely on SEO paragraphs for policy. Exclusion beats hope.
Fix the sources before you tune
Retrieval tuning without index cleanup is expensive whack-a-mole. Teams that fix sources first often find they never needed the second model pass they budgeted for.
Wrong-page answers feel like AI quirks. They are search results you can audit. Follow the citation, fix the source graph, recrawl, and test paraphrases — before you ask the model to be more careful.
