Most chatbot evaluations ask whether the system can find a page titled “Shipping” when the visitor types “shipping”. That is not a hard problem. The hard problem is the visitor who asks “how much to send a bag to France?” when your page never uses the word bag, and France only appears in a table. That gap — between the language of the page and the language of the question — is where keyword systems quietly fail and retrieval systems earn their keep.
Paraphrase testing is the shortest way to see which one you are buying. One fact, three phrasings, five minutes, no sales call. Run it on every vendor on your shortlist, including us.
What you are actually testing
You are not testing eloquence. You are testing whether the system retrieves the same underlying fact when surface vocabulary changes. Semantic search should; lexical search often will not. A chat skin does not change that arithmetic — it only makes the failure look like a polite shrug or, worse, a confident guess.
How to run it in five minutes
- Choose one fact from your published site: a price, a window, a region, a constraint. Write down the exact sentence on the page.
- Write three questions. One may share vocabulary with the page. At least one should share almost none — synonyms, customer slang, or a concrete scenario.
- Ask all three in a fresh conversation (or clear context between turns if the product allows).
- Compare the factual core of each answer, not the prose. Same number, same constraint, same exception — or not.
Example shape for a shop: the page says “Tracked shipping to EU destinations, £6”. Phrasing A: “How much is shipping to France?” Phrasing B: “Cost to send a bag to Paris?” Phrasing C: “Do you deliver to the EU and what’s the rate?” A retrieval system should land on £6 and tracked for all three. A keyword system often nails A, hesitates on B, and wanders on C.
How to score the results
| Pattern | Likely cause | What to do |
|---|---|---|
| Same fact all three times | Retrieval is doing real work | Proceed to citation and refusal tests |
| Finds it once, misses twice | Lexical match dressed as AI | Ask how indexing works — then re-test |
| Three different numbers | Weak grounding or mixed chunks | Stop; this will create chaos live |
| Invented detail on the hard phrasing | Guessing when retrieval fails | Fail the vendor on that turn |
Inconsistency across paraphrases is one of the cleanest buying signals in the category. It does not require a benchmark suite. It requires a fact you already know and the discipline not to coach the widget mid-test.
Common mistakes that invalidate the test
- Using the vendor’s demo site. Their pages are written for their demos. Use your content or a trial seeded with it.
- Asking follow-ups that leak the answer (“I mean the £6 tracked option”). You are then testing conversation memory, not retrieval.
- Scoring on tone or length. A short correct answer beats a long approximate one.
- Testing only FAQs that are already phrased as customer questions. Those pages were written to be found.
If you want the broader evaluation around this check, pair it with the outside-content question and the citation open from how to test an AI support tool. Paraphrase testing is the retrieval slice of that set.
What to ask the vendor afterward
When a triad fails, the useful conversation is operational, not theatrical. Ask what was indexed, when it was last crawled, and whether answers are constrained to retrieved passages. Ask whether they use embeddings over your pages or primarily keyword plus an LLM rewrite — the RAG vs thin-wrapper distinction shows up here in behaviour before it shows up in architecture diagrams.
Also ask what happens on miss: refusal and a human path, or a fluent attempt anyway. Paraphrase failures often travel with hallucination risk; the systems that cannot find the page are the ones tempted to invent its contents.
The shortest version
- One published fact. Three wordings. At least one with almost no shared vocabulary.
- Expect the same fact back each time — not the same sentence, the same fact.
- Treat one-in-three hits as a keyword system until proven otherwise.
- Run the triad on Matter Chat and everyone else the same afternoon.
Five minutes is enough because the failure mode is loud. You do not need statistical significance to notice that “send a bag to France” and “shipping to the EU” returned different policies. You need a notepad and the willingness to believe the widget over the pitch.
