A marketing team adds chat and reports that visitors who used it converted at four times the site average. Everybody is delighted.
Nobody asks whether people who engage with chat were already more likely to buy. They were. The chat inherited the credit for intent it did not create, and the reported figure is mostly selection rather than effect.
That does not mean chat does nothing. It means nobody at that company knows what it does, and the number on the slide is not the answer.
The first question: what are you actually measuring
Engagements, captured leads, assisted conversions or incremental conversions. These are four different numbers and only the last one tells you whether anything changed.
The selection problem, stated plainly
People who open a chat are more engaged before they open it. Comparing them to everybody else measures who they are rather than what happened. Every inflated chat figure comes from this.
How to measure it honestly
Compare comparable visitors. Same pages, same sources, same intent signals, with and without exposure. Where a proper holdout is possible, use one, because it is the only clean answer available.
What a realistic effect looks like
Meaningful on pages where questions genuinely block decisions, and close to nothing on pages where they do not. A tool reporting a uniform uplift everywhere is reporting attribution rather than effect.
Where chat actually creates conversions
Pages with unanswered practical questions. Sizing, compatibility, coverage, price, availability and eligibility. These are decisions people abandon over, which is the ground in sizing questions are killing your conversion rate.
Where it does not
Simple pages with no friction, or traffic with no intent. Adding chat to a page that was already clear produces interactions and no additional revenue.
Measure the questions, not just the conversions
The transcripts tell you what your pages fail to answer. Fixing the page is frequently worth more than the chat, and the chat is what found it.
Set up the attribution deliberately
Decide in advance whether a conversion after an engagement counts, within what window, and how it interacts with other channels, which is the ground in how to track chatbot conversion rate.
Do not let it double count
A conversion credited to chat, to paid search and to email is one conversion in three reports. Somebody has to own the definition or every channel looks successful.
Measure lead quality, not lead count
A rise in enquiries that a sales team rates as worse is not an improvement. The count is the easy number and the quality is the real one.
Measure the after hours share separately
A large share of value frequently comes from hours nobody was working. That portion is genuinely incremental, because the alternative was no response at all.
Measure what it cost, including attention
Every element on a page competes for the same visitor. A chat that distracts from a clear call to action can reduce conversion, and only a proper test will show it.
Watch the placement variable
Where and when it appears changes the result more than what it says. This is worth testing before concluding anything about chat as a concept.
Report the number you would defend
An honest, smaller, incremental figure survives scrutiny and builds trust with a finance team. An inflated one gets questioned once and then discounted permanently, which is the same discipline described in measurement questions clients ask about pr and journalist relationship questions clients ask. Run the arithmetic on the chatbot ROI calculator, see pricing, and the sector view sits under marketing agencies.