The most expensive indexing mistake we see is not a missing page. It is a staging domain sitting quietly in the source list next to production — same paths, older facts, and URLs that look legitimate enough that nobody notices until a customer screenshots the answer.
Retrieval does not know which host is “real”. It knows which passage matches the question. If staging still says free shipping over €50 and production raised the threshold last month, the assistant can cite either. Customers experience that as the company contradicting itself, not as a crawl hygiene problem.
How staging sneaks into the index
- Someone crawls the preview URL during setup because production was still behind a freeze.
- A CMS preview subdomain was linked from an old sitemap and the crawler followed it.
- An agency left a client demo host indexed after handoff.
- www, apex, and a vercel.app / netlify.app preview were all added “just in case”.
None of these feel reckless in the moment. Staging is where you test. The mistake is leaving it in the same knowledge base that serves live visitors.
What poisoned answers look like
The answer is fluent. The citation resolves. The page looks like yours. The fact is simply from the wrong generation of the site. Paraphrase testing makes this worse to spot, because one phrasing may hit production and another may hit staging — inconsistency that looks like a flaky model.
Fix it before you rewrite content
- Open the knowledge source list and sort or scan by domain.
- Remove every non-production host. Do not “pause” them and hope.
- Re-crawl production (or re-add the canonical URLs) so the index is only live pages.
- Ask the questions that previously cited staging and confirm the new citations.
- If staging must stay available for QA, put it on a separate bot with its own key.
The first-site crawl guide is explicit about using the canonical production domain for this reason. Separate bots for staging are cleaner than widening a production index to include test hosts — the same principle as domain locks for the widget itself.
Preview deploys are staging too
Pull-request previews are useful for design review and useless as knowledge sources. They are temporary, often behind auth, and frequently contain unfinished copy. Indexing them teaches the assistant unfinished truths. If you need to validate how the widget behaves on a preview, lock that origin on a staging bot — do not feed the preview’s HTML into production knowledge.
Prevention that survives handoffs
Write the allowed knowledge domains next to the allowed widget domains. Treat both lists as launch gates. When someone new joins and “just adds the site”, they should have a written list of hosts that are permitted to be crawled, not a tribal memory of which URL is safe.
Quarterly, skim citations from a sample of live conversations. You are looking for host drift as much as content gaps. Poisoning is quiet until it is public; a five-minute domain audit is cheaper than a returns-policy correction thread on social.
The short version
If answers feel “almost right but oddly outdated”, inspect domains before prompts. Staging is not a softer production — it is a competing source of truth with a higher chance of being wrong. Remove it, re-crawl the real site, and only then resume persona and tone work.
