Picture a decorator who crawls her site, asks her new assistant about prices, and is told about a spring discount she stopped offering two years ago. The page is still live. The only link to it sits in an old blog post, but the crawler follows that link, reads it, and treats it as fact.
That is the whole lesson in one example. When you train a chatbot on your website, the crawler does not know which pages you are proud of and which you forgot about. It reads what is published. If your site is tidy, you get a well-informed assistant in an afternoon. If it is not, you get an assistant that faithfully repeats your old mistakes.
This post covers what a crawl picks up, which kinds of page help or hurt, and the tidy-up worth doing before you press the button.
What a crawl actually collects
A crawler starts at the address you give it and follows the links it finds, page by page, reading the visible text. SpideyChat's crawler reads up to 150 pages per run. It is looking for words: headings, paragraphs, lists, table contents, footer details like your address and opening hours.
It does not see your site the way a visitor does. A few things are worth knowing:
- Images carry no information unless there is text. A price list uploaded as a picture is invisible. So is a banner that says "Now covering Leeds".
- Repeated text is repeated weight. A footer that says "Free quotes" on every page is not a problem. A sidebar advertising a discontinued service on every page is.
- Forgotten pages still count. An old campaign page linked from a 2021 blog post, or a draft landing page someone added to the footer, will be found and read.
- Dates are rarely obvious. A blog post from 2022 reads exactly like one from last week unless the date is in the text itself.
Once the pages are read, the assistant answers from that content. When a question has no relevant content behind it, a grounded assistant says so and takes the visitor's details instead of inventing something. That behaviour is the safety net. It does not help when the content exists but is wrong.
Which pages help and which confuse
Here is how the usual page types behave in practice.
| Page type | Effect on answers | What to do before crawling |
|---|---|---|
| Service pages | Very useful. Visitors ask about services more than anything else | Check each one names what is included and what is not |
| Pricing page | Very useful if current, harmful if stale | Update figures, add "from" wording where prices vary |
| Areas covered / locations | Useful, often thinner than you think | List towns or postcodes in text, not just a map |
| FAQ page | Useful, if the answers are still true | Delete answers that no longer apply |
| Policies (cancellation, refunds, guarantees) | Useful and often forgotten | Make sure there is one current version |
| About / team page | Mildly useful | Remove staff who have left |
| Blog posts | Mixed. Good for explaining, risky for facts | Update or unpublish posts with old prices, offers or plans |
| Campaign and offer pages | Usually harmful once expired | Unpublish or redirect them |
| Job adverts, news, press | Rarely useful, sometimes misleading | Unpublish anything out of date |
The pattern is simple. Pages that describe how your business works today help. Pages that describe a moment in time, a promotion, an announcement, an opinion, confuse.
Blog posts deserve a note of their own. A good explainer about how you prepare walls before painting is excellent material. A post titled "Why we are thinking about offering wallpapering next year" will, sooner or later, lead to a visitor being told you offer wallpapering.
The hour of tidying that saves a week of corrections
You do not need a redesign. You need a pass through the site with one question in mind: if a customer read this page today, would anything on it be untrue?
- List your pages. Your website platform can usually export them, or you can open your sitemap. Count them. If you are over 150, decide now what to leave out.
- Unpublish expired offers. Anything with a deadline that has passed goes. If you cannot unpublish it, add a clear line at the top saying the offer has ended.
- Fix the pricing page first. It is the page visitors ask about most, and the one where a wrong answer costs you money or goodwill.
- Put key facts in text. Opening hours, areas covered, minimum job size and contact routes should be written as words somewhere, not only shown in images or embedded maps.
- Remove contradictions. If the FAQ says "call-outs within 48 hours" and the service page says "same week", pick one and change the other.
- Decide on the blog. Keep the explainers, update or unpublish the rest.
- Crawl, then test. More on that below.
If your site is in genuinely poor shape, there is a separate guide to training a chatbot on a messy website that goes further.
What the website cannot tell it
Even a well-kept site leaves gaps, because websites are written to persuade, not to answer awkward questions. Most sites do not say what happens if a customer cancels the day before, whether you work weekends at a premium, or which jobs you turn down.
Those answers usually live in your head, your email templates or a document on someone's laptop. The crawl is one of four ways to teach the assistant. You can also describe the business in plain language, upload documents such as a price list or terms, and add Q&A pairs for the specific questions your website skips. A crawl plus a dozen well-written Q&A pairs covers far more real enquiries than a crawl alone.
Checking the crawl did what you think
Do not assume. Open the assistant yourself and ask the questions you actually get. A quick routine that works:
You: Do you do exterior painting in Harrogate? Assistant: Yes, we cover Harrogate for exterior work. We usually need to see the property before giving a price. Would you like to leave your details so we can arrange a visit?
Then open the service page and the areas page and confirm the answer matches. Do that for your ten most common questions. Any answer that is wrong points to a page that needs fixing, not an assistant that needs scolding. Fix the page, re-crawl, ask again.
Also ask two or three questions you know the site does not cover. You want to see the assistant admit it does not have that information and offer to take details. If it guesses, something on the site is implying an answer, and it is worth finding.
What to do this week
Pick a quiet hour. Open your sitemap and go through the list with the table above beside you, marking each page keep, fix or remove. Unpublish expired offers and correct the pricing page before anything else.
Then run the crawl, ask your ten common questions and two awkward ones, and note which answers were wrong. Each wrong answer is a page to edit. Over the following weeks, the unanswered questions report will show you what your website never said; our post on turning unanswered questions into website pages covers what to do with it. If you want the technical detail of starting a crawl, the docs walk through it, and you can try it on your own site from the free plan.