We run our own agent on spideychat.com. It is the most useful testing we do, mostly because it embarrasses us.
Here is what we got wrong. None of it is exotic — that is rather the point.
We did not read the conversations
For about a fortnight we watched the numbers and not the transcripts.
Everything we eventually fixed was sitting in those conversations from day one, written in the visitor's own words. The dashboard said "conversations: up". The transcripts said "three people in a row asked whether it works on Webflow and got a vague answer".
Read the transcripts. It is twenty minutes and it is the whole game.
We had contradictions on our own site
We found prices in three places that did not agree, and a page describing a feature by an old name.
The agent was not wrong; it was picking whichever version retrieval surfaced. Fixing our content fixed the answers, and the site got better for humans at the same time.
We turned lead capture on too late
For the first few weeks the agent had good conversations and recorded nothing.
We could see people asking buying questions at 11pm — and we had no way to follow any of them up. Those were real, and we lost them to a configuration switch we had not flicked.
Turn it on in week one. Point it somewhere you look.
Our first welcome message was useless
"Hi! How can I help?" It gave the visitor nothing and put all the work on them.
Naming the two or three things it can actually do changed how many people engaged, immediately.
We over-thought the persona and under-thought the content
We spent real time on tone and almost none on whether the answers were correct.
The tone made no measurable difference. The content made all of it. Nobody has ever complained that an assistant was insufficiently charming; they complain that it did not know something.
The thing that worked immediately
Quick-action pills. Most visitors tap rather than type, and the pills tell them what the thing can do before they have to guess.
Changing them from what we offer to what a customer wants was a five-minute edit with a visible effect.
The uncomfortable lesson
Almost none of our problems were AI problems. They were content problems, configuration problems, and not-looking problems.
That is good news, because all three are fixable in an afternoon — and unlike the model, they are entirely under your control.
Related: your first hour with SpideyChat · reading your conversations.