Adding a website assistant is easy to do and easy to measure badly. A dashboard showing thousands of conversations looks impressive. It says nothing about whether customers got good answers, whether support load dropped, or whether trials converted.
This checklist covers the chatbot metrics worth tracking for a SaaS company, with a note on why each matters.
1. Answer accuracy
What: the share of answers that were correct and useful, based on a regular sample you review by hand.
Why: everything else depends on it. A fast wrong answer is worse than no answer.
2. Satisfaction ratings
What: thumbs up or down on individual answers.
Why: direct feedback from users, and an early warning when content goes out of date.
3. Escalation rate
What: how often conversations are handed to a person.
Why: too high means the docs are thin; too low may mean users cannot reach a person when they need to.
4. Unanswered questions
What: questions the assistant could not answer.
Why: this is your documentation backlog, and one of the most valuable outputs. The process of using it is covered in choosing a support assistant for a software product.
5. Ticket volume
What: support tickets per week, compared with before launch.
Why: the clearest sign of reduced workload. Compare like with like, since volume also changes with user growth.
Want a 24/7 AI Employee for Your Website?
See how SpideyChat can answer customer questions, capture leads and handle enquiries while your team is offline.
6. Time to first answer
What: how quickly users get a first reply, split by working hours and out of hours.
Why: out-of-hours improvement is often the biggest gain, as covered in a website assistant for out-of-hours SaaS visitors.
7. Leads captured
What: qualified sales leads and trial signups that came through conversations.
Why: shows the assistant's value beyond support.
8. Trial conversion for users who asked questions
What: conversion rate of trial users who used the assistant, compared with those who did not.
Why: a direct link between help and revenue, though be careful: engaged users convert better anyway.
9. Questions by page
What: which pages generate the most questions.
Why: a page that produces many questions is often unclear. Pricing and setup pages are the usual suspects. Fixing the page reduces questions for everyone, including visitors who never ask.
10. Returning users who ask again
What: users who ask the same question more than once.
Why: a sign the first answer did not work. Review these conversations and improve the underlying content. Often the answer was technically right but skipped a step that new users need, such as where a setting lives or what to do after the first step succeeds.
What to ignore
Total conversations measures activity, not value. Raw deflection rate can be raised simply by making support hard to reach. Average conversation length has no clear good direction.
Setting a baseline
Measure ticket volume, first reply times and trial conversion for a month before launch. Without a baseline, you are guessing. Faster handling times for your team are one knock-on effect worth tracking, explored in why average handle time drops when you add an assistant.
SpideyChat for SaaS companies reports conversations, ratings, leads, response times and unanswered questions. The installation guide and training guide cover setup.
Retailers measuring the same thing tend to focus on the sales side, covered in choosing a website assistant for an online store. Whatever the industry, the principle holds: measure outcomes, not activity.