Teams celebrate a docs rewrite when the new pages are published. If nobody triggers a recrawl, the support bot is still answering from the old index. Visitors get confident answers that match last quarter’s policy, complete with citations to URLs that now redirect or contradict the page they replaced.
This is not a model failure. It is a maintenance gap that looks like one — because the assistant speaks fluently while the knowledge layer is stale. Treat recrawl as part of the launch checklist for any content change that matters to support, not as an optional cleanup you do when someone notices.
Why “published” and “indexed” diverge
Retrieval systems do not watch your CMS. They read what was crawled last — a snapshot of URLs, chunked and embedded at a point in time. When you rename slugs, merge articles, or move from Zendesk to a custom help centre, the public site updates immediately. The index updates only when something asks it to look again.
- Redirects can leave old chunks alive if the crawler never follows them to the canonical page.
- Draft-to-live flips do not remove superseded pages until they 404 or are excluded.
- PDFs and attachments often sit outside the crawl scope you set at launch.
- Staging previews accidentally indexed months ago can still win retrieval if nobody pruned them.
What to recrawl — and what to delete
A full recrawl after a major rewrite is the safe default. It is not always the fastest fix. If you know exactly which URLs changed, targeted recrawl of those paths plus explicit removal of retired URLs can shorten the window where two versions compete.
Deletion matters as much as refresh. Leaving dead pages in the index is how you get two return windows, two pricing tables, or two definitions of “enterprise plan”. Before you recrawl, list URLs that should no longer exist as sources and remove them from the bot’s scope. The guide to crawling your first site covers exclusion patterns; the same rules apply in reverse when content retires.
A verification pass that catches stale citations
Do not trust the crawl job’s green checkmark alone. Run a short regression on questions your inbox already knows the new docs answer. For each answer, open the citation. You are checking three things: the URL is the one you intend, the claim on the page matches what the bot said, and no second citation from an old path still appears on a paraphrase.
- Pick ten questions the rewrite was supposed to fix.
- Ask each in two phrasings — page title language and how customers actually type.
- Record the citation URL and whether the sentence is still on that page.
- Flag any answer that cites a redirect, an archive, or a page you thought you removed.
- Fix indexing or content before you announce the bot is “updated”.
If the bot answers correctly but cites the wrong page, you have a content gap dressed up as success. The next visitor who phrases the question differently will get the old chunk. Fix the index, not the prompt.
Fold recrawl into how you ship docs
One-off recrawls after big projects fail because the next small edit does not trigger one. Assign an owner: whoever merges the docs PR also confirms the bot’s sources were refreshed or scheduled. For teams without a dedicated ops role, tie recrawl to the same ticket that closes the docs project.
- Major rewrite: full recrawl, remove retired URLs, run the ten-question regression.
- Single policy page: targeted recrawl of that URL and anything that duplicated the policy.
- Pricing change: recrawl pricing and billing FAQ the same day numbers go live.
- Rename or merge: set up redirects, then recrawl targets and purge old paths.
When to tell visitors the bot may be wrong
During a large migration, a temporary banner on the widget — “We updated our help centre; answers may not reflect every change yet” — is honest product design. It beats silent wrong answers while the index catches up. Turn it off only after the verification pass passes, not when the CMS publish button was clicked.
“The rewrite is not done for support until the thing that answers customers has read the new pages.”
Teams that ship docs weekly sometimes schedule recrawl on the same cadence (nightly, or after every merge to main) so the index never lags more than a business day. That is overkill for a five-page FAQ that changes twice a year; it is reasonable when pricing and compliance copy move with every release train.
Recrawl is boring infrastructure. It is also the difference between a bot that tracks your docs and one that preserves your mistakes in vector form. Schedule it like deploys, verify it like tests, and treat stale citations as bugs — because that is what they are.
