Buyer's Guides· 7 min read

How to Evaluate an AI Chatbot in One Afternoon

A practical test plan that surfaces the real differences between products faster than any feature comparison.


Feature comparison tables are close to useless here, because every product lists the same twenty features and you will use six of them.

Here is a test plan that surfaces the real differences in an afternoon.

Before you touch any software (30 minutes)

Write down the ten questions your customers actually ask. Get these from whoever answers the phone, not from your own assumptions. The gap between the two is usually instructive on its own.

Write down what should happen when it cannot answer. Capture a lead? Open a ticket? Fetch a person? This decides more than any feature.

Write down your peak month volume. You will need it to price anything honestly.

The test (60 minutes, per product)

1. Train it on your real website. Not a demo site. If a product cannot ingest your site quickly, that is a finding.

2. Ask your ten questions. Score each: correct, vague, or wrong. Vague is a content problem. Wrong is a product problem.

3. Try to break it. Ask a price you do not publish, a service you do not offer, a location you do not cover. Correct behaviour is a clear admission plus a route forward. Anything invented disqualifies it.

4. Ask for a human. Say "I want to speak to a person". It should do it immediately, without arguing. Bots tuned for containment often resist, and that is what your angriest customer will meet.

5. Check what exists afterwards. Is there a lead? A ticket? A notification? Or just a transcript? This is the chatbot-versus-employee distinction and it decides whether the thing produces anything.

6. Look at it on a phone. Most of your visitors will.

The pricing check (15 minutes)

Price your peak month, not your average, on each product. Include extras: additional bots, seats, branding removal, integrations.

If a vendor cannot give you a number for a stated volume, that is also a finding.

Scoring

Weight it like this:

Most feature comparisons invert this, which is why they lead people to the wrong product.

Start here

Build one from your website URL, free and without signing up, and run the ten questions. Whatever else you evaluate, you will have a baseline within a minute.

Related: best AI chatbot for small business · how to stop it inventing answers.

Frequently asked questions

What is the fastest way to compare products?
Point each at your own website and ask the same five questions. Differences that matter appear within minutes; feature tables hide them.
What should I test first?
The failure mode. Ask something your site does not cover and see what it does. That behaviour is what customers meet on their worst day.
How long does a proper evaluation take?
An afternoon. Most of it is deciding what you want it to do, not testing the software.
Should I involve my team?
Have whoever answers enquiries write the test questions. They know what customers actually ask, which is rarely what the owner assumes.

Keep reading

How to Evaluate an AI Chatbot in One Afternoon · SpideyChat