A support team reports a seventy per cent deflection rate after three months. The number is quoted upwards and everybody is pleased.
Within that seventy per cent are customers who got an answer, customers who gave up, customers who found the answer elsewhere afterwards and customers who complained publicly instead.
Nobody separated them, because the tool reports one number and the number looks good. The team is now managing towards a metric that cannot distinguish success from abandonment.
The first question: what was the baseline
Volume, mix, response times and satisfaction before anything changed. Teams that did not capture this spend the following year arguing about whether it helped.
Why deflection is the wrong headline number
It counts conversations that did not reach a person, which includes every failure mode as well as every success. A customer who left angrily is counted identically to one who was helped.
Measure resolution instead
Whether the customer's issue was actually settled. This requires asking them, or following what happened next, and it is the only number that means anything on its own.
Measure what happened in the next twenty four hours
Did they come back, contact another channel, leave a review or abandon a purchase. Repeat contact is the clearest signal that a deflection was not a resolution.
Measure satisfaction on automated conversations specifically
Separately from human conversations. A single blended score hides whichever of the two is failing.
Measure the mix reaching the human team
If the routine questions have gone and the complex ones remain, handling time will rise and that is a success rather than a problem. Teams frequently misread this as a decline.
Measure the after hours contribution separately
Everything answered when nobody was working is incremental. It is also the easiest part of the case to defend to a finance team.
Measure escalation quality, not just rate
How many handovers arrived with full context, and how often the customer had to repeat themselves, which is the ground in handover questions between bot and agent.
Measure the coverage gaps
The questions that were asked and not answered are the most valuable output of the whole system, and they only appear if somebody reads the transcripts.
Measure what it freed the team to do
Backlog cleared, proactive work started, training completed. If the capacity was released and immediately absorbed invisibly, nobody will believe it happened.
Do not measure cost per contact alone
It improves automatically when volume shifts, regardless of whether anybody was helped. On its own it invites the wrong decisions.
Give it long enough to be real
A few weeks of stable operation, after the initial tuning, before drawing conclusions. Early numbers reflect configuration rather than performance.
Compare against the right alternative
Not against perfect service, but against what actually happened before, which for most weekend and evening contact was nothing at all, as set out in weekend and holiday cover questions.
Report a number you would defend under scrutiny
An honest, smaller, well defined figure survives a finance review. The before and after picture in a week of saas support before and after an assistant and the metric selection in the customer support metrics that actually matter now are the right starting points. See pricing, run the arithmetic on the chatbot ROI calculator, and the sector view sits under small business.