A chatbot should learn from the pages that state facts your customers ask about: what you sell, what it costs, how delivery and returns work, when you are open and how to reach you. It should leave out anything that is out of date, repeated elsewhere, or written to market rather than to inform, such as expired promotions and most blog posts. Get that split right and the bot answers more accurately from fewer pages.
Our main guide on how to train your chatbot on your website content covers the whole training process. This post deals with one decision inside it: which pages go in.
Why page choice matters more than page count
A chatbot treats everything it learns as true. It has no way of knowing that the "Summer sale: 20% off all installs" page ended in August, or that your 2022 blog post about pricing was superseded. When a visitor asks a question, the bot searches its knowledge for the closest passages and answers from them. If an old page is a close match, it gets used.
So every page you add is a small bet. A good page makes some answers better. A stale or contradictory page makes some answers worse, and you will not know which until a customer quotes it back to you.
Pages to include
Start with the pages that hold facts customers act on.
- Service and product pages. What you do, who it is for, what is included.
- Pricing. Even if prices vary, the page explaining how pricing works answers a lot of questions.
- Delivery, returns, cancellations and warranties. These attract the most specific questions and the most frustration when answered wrong.
- Hours, locations and contact details. Small pages, asked about constantly.
- Your FAQ page. Already written in question and answer form, which suits a chatbot well.
- About and team pages, if visitors ask who they will be dealing with.
- Policy pages that customers ask about, such as privacy or booking terms, when they hold practical detail rather than pure legal boilerplate.
If you want a quick sense of how a bot answers from just your core pages, paste your address into the free trial builder. It reads up to ten public pages and gives you an assistant to question straight away.
Pages to leave out
Expired promotions and seasonal pages. The most common source of confidently wrong answers. If the offer is over, unpublish the page or remove it from the bot's knowledge.
Most blog posts. A blog post is written to attract and persuade, often discusses options you do not offer, and dates quickly. The SpideyChat crawler skips blog, news, press, tag, category, author and archive URLs by default, along with date-stamped permalinks, and does so before applying the page limit so posts cannot crowd out your service pages. There is a tickbox to include them if your blog genuinely holds how-to answers customers need.
Near-duplicate pages. Location pages that differ by one town name, print versions, filtered product listings and old copies of a page under a different URL. The crawler removes exact duplicate URLs, but it cannot tell that two different URLs say the same thing. Duplicates waste your page allowance and, if they disagree slightly, give the bot two answers to choose from.
Thin pages. Thank-you pages, login screens and pages that are mostly an image. Pages with almost no text are skipped automatically, but pages with a single line of filler still take up room.
Pages you would not want quoted. Internal pricing for trade partners, careers pages if you do not want the bot discussing salaries, and anything legal that you would rather a person explained.
Planning around the crawl limit
On Starter and above, the crawler reads up to 150 pages per run. On the Free plan, setup reads 10 pages once, and after that you add knowledge by pasting text, adding Q&A pairs or uploading files. The pricing page has the full plan breakdown.
The crawler finds pages through your sitemap.xml first and falls back to the links on your homepage if there is no sitemap. That has a practical consequence: if your sitemap lists 600 URLs, it will take the first 150 it finds after leaving out blog and news URLs, which may not be the 150 you care about.
You can steer this. The crawl box accepts a sitemap address directly, so if your site generates separate sitemaps (many WordPress SEO plugins produce one for pages and another for products or posts), point the crawler at the one that holds your core pages. For a large store, it often works better to crawl the page sitemap and upload a product catalogue as a CSV or document than to crawl hundreds of near-identical product URLs.
Two other limits are worth knowing. The crawler strips navigation, headers, footers and sidebars from each page, so facts that only appear in your footer, such as a phone number or opening hours, should also appear in the body of a contact page. And very long pages are cut off after roughly 24,000 characters of text, so a huge single-page FAQ may lose its last sections. Split it, or upload it as a document.
Linked PDFs are not followed by the crawler. If your price list or brochure is a PDF, upload it directly; PDF, DOCX, TXT, CSV and Markdown are all accepted.
After the crawl: prune, then test
When a crawl finishes, the dashboard tells you how many pages were learned, how many were skipped as unreachable or empty, and how many blog or news pages were left out. Each crawled page then appears as its own source, named by its path, which you can open, edit or delete.
Spend ten minutes going down that list. Delete the expired offer you forgot about, the duplicate location pages and anything you would not want quoted. Then ask the questions customers really ask, as described in how to test your chatbot before going live. If an answer is wrong, find the page it came from and fix the page or remove it.
When your site changes later, update the knowledge to match. If you re-crawl after a big rewrite, delete the old sources for the pages that changed first, so the bot is not holding two versions of the same page. Questions the bot could not answer well are logged as knowledge gaps, which tell you which page you are missing.
A simple order of work
List your 20 most-asked customer questions. Make sure each one is answered on a page you will include, writing a short page or a Q&A pair where one is missing. Crawl, prune the source list, and test with those 20 questions. The training section of the docs walks through each knowledge option if you want the button-by-button version.