Every vendor demo goes well. The questions are picked in advance, the knowledge base is tidy, and the person driving knows which buttons not to press. You leave thinking the thing is clever, and you still have no idea how it will behave at 11pm when a customer asks something your website never covered.
Choosing an AI receptionist is less about features and more about behaviour under pressure: what it says when it does not know, how it gets a human involved, what happens to your data and what the bill does in a busy month. Those are the places products differ, and they are rarely on the slide deck.
Below are twelve questions to put to any vendor, grouped by the risk they cover. For each one there is a note on what a good answer sounds like. Take them into the call, or send them by email and see how the replies come back.
First, what is it and where does it work
1. Which channel does it actually cover? The phrase "AI receptionist" gets used for two different products. Some answer phone calls with a voice. Others, SpideyChat included, sit on your website and handle visitors in a chat window. Neither is wrong, but you need to know which one you are buying before anything else makes sense.
A good answer is one sentence and names the channel. A worrying answer is "it handles all your customer communication" with no detail.
2. How do I teach it about my business, and how long does that take? Ask them to show you, not describe it. Common routes are crawling your site, uploading documents, writing question and answer pairs or describing the business in plain language.
A good answer includes a limit, such as a page cap on the crawler or the file types accepted. Limits mean someone has thought about it.
Accuracy and what happens at the edge
This is the section that matters most, and the one demos avoid.
3. Where does each answer come from? You want to hear that replies are drawn from the content you supplied, not from the model's general knowledge of the world. Our guide to grounding explains why this is the whole game.
4. What does it say when it has nothing relevant? Ask them to type a question the knowledge base cannot answer, live. A good assistant says it does not have that information and offers to take the visitor's details. A bad one produces a confident, plausible and invented reply.
5. Can I see the questions it could not answer? Unanswered questions are the most useful report you will get, because they tell you what to write next. If the product does not record them, you are flying blind.
Getting a person involved
6. How does a conversation reach a human, in practice? "It escalates" is not an answer. Ask what the visitor sees, what your team receives and whether someone can take over a live chat from an inbox.
7. What happens to that handover outside working hours? A transfer to a human at 2am is only useful if it becomes a captured enquiry or a numbered ticket that someone picks up in the morning. Ask where it lands. Our piece on when to hand over covers the triggers worth setting.
8. Which actions are on by default? Lead capture, tickets, webhooks and transfers are powerful. A careful product has them switched off until you choose to turn them on, per assistant.
Data and security
9. Where is conversation data stored, and for how long? A good answer mentions encryption in transit, how stored credentials are protected, whether your data is isolated from other customers and whether you can set a retention window.
10. Which certifications do you hold? Ask it plainly. If they have none, the honest reply says so. Be wary of vague phrases like "enterprise-grade security" with no document behind them.
Money and leaving
11. What does my bill look like in my busiest month? Pricing per conversation or per resolution can look cheap in a quiet month and hurt in a good one. Ask for a worked figure at double your normal traffic. We go through the models in pricing models explained.
12. How do I leave? Ask about exporting leads and transcripts, deleting data and whether there is a notice period or annual lock-in. A vendor confident in the product answers this without flinching.
Telling good answers from evasive ones
After a few calls the answers blur together. Score them straight after each conversation using something like this.
| Question area | A straight answer | A reason to worry |
|---|---|---|
| Channel | Names website, phone or both | Talks about "all channels" |
| No relevant content | Says so, captures the enquiry | Changes the subject, or demos only easy questions |
| Handover | Shows the inbox and the takeover | Says "it escalates" and moves on |
| Actions | Off until you enable them | Everything live from day one |
| Certifications | Names what they hold, or says none | "Bank-level security" with no detail |
| Busy month bill | Gives a number | Needs to "speak to the team" |
| Leaving | Explains export and cancellation | Mentions only annual discounts |
A few habits make the scoring more honest:
- Ask the same question to every vendor, word for word. Otherwise you compare their best answers to different questions.
- Write the reply down within ten minutes. Memory flatters whoever was most charming.
- Count the times you heard "it depends". Sometimes it genuinely does. Four times in one call is a pattern.
- Note what they volunteered. A vendor who tells you a limit before you find it is one you can work with.
What to do before your next demo
Pick three real enquiries your business received last month, including one awkward one that your website does not cover. Send them to the vendor before the call and ask them to run those, not their own examples. That single request changes the tone of most demos.
Then check the pricing page yourself rather than relying on the call. On ours, the pricing is flat monthly, so you can work out the busy-month figure without asking anyone.
Finally, do not sign anything until you have tested the product with your own content. Our free trial test plan sets out how to do that in an afternoon, with a scoring sheet you can reuse for every product on your shortlist.