Crawler
A crawler follows links. Anything not linked from somewhere it reaches effectively does not exist.
Why it matters
Crawl coverage is the first thing to check when an assistant refuses something it should know. Orphan pages, JavaScript-only navigation, and robots directives all silently remove content from what it can answer.
- Pages missing from the index with no obvious reason
- Content reachable only through a search box, never a link
- Tag archives and pagination indexed instead of real articles
How discovery works
It starts at a URL and follows links outward. Pages linked from nowhere — orphans — are never found, which is why an important page missing from the index is usually a site navigation problem rather than a crawler one.
What to exclude
Tag archives, paginated listings, author pages, and search-result URLs add volume without adding answers, and they dilute retrieval by competing with real content. Reviewing the source list after the first crawl is worth the few minutes.
Keeping it current
Scheduled re-crawls pick up changes and re-index only what actually changed, so published edits reach the assistant without anyone triggering anything manually.
Common questions
- Why is a page missing from the index?
- Usually one of three reasons: nothing links to it, it is blocked by robots rules, or its content renders only via JavaScript the crawler did not execute. Check in that order.
- How often does it re-crawl?
- On a schedule, re-indexing only what changed. For a page you have just published and need answerable now, a manual re-crawl is faster than waiting.
From the blog
All posts- MeasuringDeflection rate is the most overstated number in support automationCounting bot conversations as deflected tickets overstates the result. The honest version is a before-and-after on ticket volume, read next to satisfaction.Read
- EvaluatingHow to test an AI support tool before you trust itEvery AI support tool demos well, because demos ask questions the content covers. Four questions that separate them, and what a good answer looks like.Read
- MeasuringResolution rate vs deflection: stop mixing the twoResolution and deflection answer different questions. Mixing them inflates the result and hides whether customers actually got what they needed.Read
Answer honestly. Capture the rest.
Point Matter Chat at your site and see what it can — and can't — answer. It's honest about both.
No credit card. 2 minute setup.
Every answer cites the source it came from. When there isn't one, it says so — and hands the visitor to you.
Installs on the tools you already run.