Every assistant in this category demos well. That is not a conspiracy, it is arithmetic: a demo is built from questions the content covers, and any retrieval system answers those. The differences only show up at the edges — when the question is slightly outside the material, or the answer matters enough that being confidently wrong is expensive.
So the useful evaluation is not a feature comparison. It is four questions, asked of a live widget, with a note of what came back. It takes about twenty minutes per vendor and it does not need a sales call. Run it on us too — if we come out badly on your content, we would rather you found out now.
1. Ask something your site does not cover
This is the one that matters most, and it is the one nobody demos. Pick a question that is plausible for your business but genuinely absent from your published pages. For a shop: “do you offer gift wrapping?” when you do not. For a clinic: something clinical. For a SaaS product: a feature on nobody’s roadmap.
There are only two acceptable answers. The assistant says it does not have that information and offers a person, or it says nothing useful and offers a person. What you are watching for is the third behaviour: a fluent, plausible, entirely invented answer, delivered in the same confident register as the true ones.
2. Ask the same thing three different ways
Take one fact from your site and ask for it in three phrasings, at least one of which shares no vocabulary with the page it lives on. If your shipping page is titled “Shipping”, ask “how much to send a bag to France?”. If your pricing page says “seats”, ask about “extra logins”.
A keyword system will find one of the three. A retrieval system should find all three and return the same fact each time. Inconsistency across paraphrases is the clearest signal that what you are looking at is search with a chat interface bolted on.
3. Follow the citation
If an answer carries a source, open it. Twice. You are checking two separate things and they fail independently: that the link resolves to a real page on your site, and that the page actually contains the claim the answer made.
A citation that points at your homepage, or at a page that vaguely relates to the topic, is decoration. It looks like accountability and provides none. The point of a source is that a reader who doubts the answer can settle it in one click — if they cannot, the citation is doing nothing except lending confidence to something unverified.
| Test | Good | Bad |
|---|---|---|
| Outside the content | Declines plainly, offers a person | A fluent answer you cannot source |
| Three paraphrases | Same fact all three times | Finds it once, misses twice |
| Follow the citation | Resolves, and contains the claim | Points at the homepage |
| Ask for a human | Hands over immediately | Tries two more times first |
4. Ask for a human
Type “I want to speak to someone”. Count how many turns it takes to get there. One is correct. Two is tolerable. Anything more is a product decision someone made deliberately, and it is the decision that turns automation into the thing customers complain about publicly.
While you are there, check what the handover carries. An escalation that arrives as a name and an email address has thrown away the most useful part of the conversation — what the customer already asked, and what the assistant already ruled out.
Why the headline number does not help
Most vendors, us included, can show you an answer rate. It is the least informative number in the category, because it is trivially improved by being less careful. An assistant that answers everything scores perfectly and tells you nothing about whether the answers were right.
% of conversations
Illustrative, not measured: both systems post the same answered and declined split, so neither number separates them. What separates them is what those declines contained.
Two systems with an identical answer rate can be completely different products. The one that matters is what happened in the 8% — whether those were genuine gaps in the content, honestly declined, or questions the system should have refused and answered anyway. Read the content gap report alongside the rate, or read neither.
The shortest version
- Ask something outside your content. Watch for an invented answer.
- Ask one fact three ways. Watch for inconsistency.
- Open a citation. Check it contains the claim.
- Ask for a human. Count the turns.
None of this requires a trial, a call, or a spreadsheet of features. It requires twenty minutes and a question you already know the answer to. If a vendor’s assistant is not on their own site for you to test, that is itself a result.
