Pages go missing from a chatbot's knowledge for a handful of mechanical reasons: the crawl hit its page cap, the crawler couldn't find your sitemap, the page builds its text with JavaScript, the content sits behind a login, or a firewall turned the crawler away. Each one leaves a clue in the crawl result, and each has a direct workaround. Start by reading the message the crawl gave you, then work through the causes below.
This is the troubleshooting companion to how to train your chatbot on your website content, which covers the full setup. Here we only deal with the pages that didn't make it.
Read the crawl result first
After a crawl, SpideyChat shows a line such as "Trained on 38 of 42 pages (212 chunks). Skipped 4 unreachable/empty pages. Left out 61 blog/news pages." Each number points somewhere different:
- Found is lower than you expected: the crawler didn't discover the pages at all. See the sitemap and cap sections.
- Skipped is above zero: pages were found but returned an error, timed out, weren't web pages, or had almost no text. See JavaScript, logins and firewalls.
- Blog/news pages left out: posts were set aside on purpose. Tick "Also learn blog / news posts" and crawl again if you want them.
To check what was learned for a specific page, open your chatbot's page, go to Manage knowledge and click "Show sources". Crawled pages are listed by path, such as /delivery. If the path isn't there, the page wasn't learned.
How the crawler finds pages
The crawler looks for /sitemap.xml, then /sitemap_index.xml. If it finds a sitemap index, it walks the child sitemaps in order until it has enough addresses. With no sitemap, it reads your homepage and follows the links on it to pages on the same domain. It doesn't go further than that one level.
That creates three common gaps:
- Your sitemap has another name. WordPress's built-in sitemap, for example, lives at
/wp-sitemap.xml. Paste the full sitemap address into the "Website or sitemap URL" box and the crawler uses it directly. - No sitemap, and the page isn't linked from the homepage. A returns policy linked only from the checkout won't be found. Add a sitemap, or link the page from the homepage. Without a sitemap, you can also enter the page that links to it (the checkout, say) as the crawl address: the crawler learns that page and the pages it links to.
- The sitemap index lists posts or products first. The crawler stops collecting once it reaches the page cap, so if the first child sitemap holds 300 products, your service pages in the third child may never be reached. Point the crawler at the child sitemap that lists your main pages (often named something like
page-sitemap.xml), then run a second crawl for the rest if you need it.
The page cap and your plan
Each crawl learns up to 150 pages on Starter and above. Blog and news addresses are filtered out before the cap is applied, so they don't crowd out your service pages. On the Free plan, the crawler runs once as a 10-page setup for your first chatbot, so a larger site will be missing most pages until you add them another way: paste text, add Q&A pairs, upload documents, or move to a plan with the full crawler. The pricing page lists what each plan includes.
For a site with more than 150 useful pages, crawl in sections by pointing the crawler at separate child sitemaps. Re-crawling a page replaces that page's older entries, so overlapping runs don't leave duplicates of the same page.
Pages that build their text with JavaScript
The crawler downloads each page's HTML and reads the text in it. It doesn't run JavaScript. Most WordPress, Squarespace and Webflow pages deliver their text in the HTML, so they read fine. Some sites, especially single-page apps and pages that load prices or menus from a separate data request after the page opens, send an almost empty HTML shell and fill it in later in the browser.
A quick check: open the page, right-click and choose View Page Source, then search for a sentence you can see on screen. If it isn't in the source, the crawler can't see it either. Pages with fewer than 120 characters of text after cleaning are skipped as empty, which is what most of these shells come to.
The fix is either to have your developer render that content on the server, or to copy the text into the bot directly using Paste text or a Q&A pair.
Content the crawler strips on purpose
To stop menus and footers repeating on every page, the crawler removes nav, header, footer and aside sections before learning a page. It also ignores scripts, embedded iframes (such as maps or booking widgets) and images. So:
- Opening hours, a phone number or a service area that appear only in the footer won't be learned. Put them on your contact page in the main content, or add them as a Q&A pair.
- Text inside images, banners and infographics is invisible. Write the key facts out as text on the page.
- Very long pages are cut at roughly 24,000 characters, so the end of a giant single-page FAQ may be lost. Splitting it into topic pages fixes that and helps answers too.
Logins, firewalls and other refusals
The crawler visits as an anonymous visitor. Anything behind a member login, a password-protected page or a "coming soon" mode returns the login screen, not your content. If customers ask about things that are only in a members area, paste the relevant text in yourself and leave out anything private.
Firewalls and bot protection can also refuse the crawler, especially on sites with aggressive security plugins or "challenge" pages for automated traffic. The crawler identifies itself as SpideyChatBot in its user agent, so you can allow it in your security settings while you crawl. Each page also has an 8-second time limit, so a very slow page can be skipped on one run and learned on the next.
PDFs and other files
The crawler learns web pages only. It doesn't follow links to PDFs, and if a sitemap lists a PDF, it skips it. Upload the file under Upload files instead: PDF, DOCX, TXT, CSV and Markdown are supported, and the text is extracted directly. Check our docs on training your bot for the upload steps.
A five-minute check after every crawl
- Read the crawl result line and note found, learned and skipped.
- Open Manage knowledge, show sources, and confirm your five most important paths are listed: pricing, services, delivery or service area, policies and contact.
- Ask the chatbot one question that only each of those pages answers.
- For any miss, find the cause above and fix it, then crawl again.
If you're comparing plans because of the page cap, the pricing page shows crawler limits, website limits and reply allowances side by side, so you can pick the plan that fits how many pages you actually need the bot to learn.