Training a Chatbot on Webflow CMS Collections

A website chatbot learns your Webflow CMS content the way a visitor sees it: from the published Collection pages listed in your sitemap. To train it well, publish items to your live domain, keep Webflow's auto-generated sitemap switched on, decide which Collections are worth learning, and re-crawl after each batch of new items. With SpideyChat, each crawl reads up to 150 pages on Starter and above, starting from your sitemap.xml.

Installing the widget on Webflow is covered in how to add an AI chatbot to a Webflow site. This guide is about the training side, which is where most Webflow setups go wrong.

How the crawler finds Collection pages

Webflow creates a template page for every Collection, and every published item gets its own address built from the Collection URL and the item's slug. Those are ordinary pages to a crawler.

SpideyChat finds them in this order:

1. Your sitemap. It reads sitemap.xml on the address you enter, following sitemap index files if there are any.
2. Links, if there's no sitemap. It falls back to the links on the page you entered, one level deep. CMS items that are only linked from a paginated Collection list, or from other item pages, can be missed this way.

So the sitemap matters. Webflow's help center says its auto-generated sitemap updates each time you publish. You switch it on in Site settings > SEO > Sitemap by setting Auto-generate sitemap to Yes, saving and publishing. If it's off and you haven't added a custom sitemap, the crawler only sees what your homepage links to.

Webflow also lets you remove a single CMS item from the sitemap with its Sitemap indexing toggle in the item's details. An item excluded that way won't be found by a sitemap-based crawl either. That's useful for items you don't want learned, but remember it also hides them from search engines.

Choose which Collections to teach

Not every Collection helps an assistant answer questions. Sort yours into two groups.

Usually worth learning: services, locations or service areas, pricing tiers, FAQs, product or feature pages, team pages with real detail, case studies, documentation.

Usually not: blog posts written for search traffic, news, press releases, event announcements that have passed.

SpideyChat makes the second group opt-in. It skips addresses that look like blog or news pages by default, before applying the 150-page cap, so posts can't crowd out the pages that answer questions. Paths containing segments such as /blog/, /news/, /posts/, /articles/, /insights/, /updates/ or /stories/ count, as do dated addresses.

Webflow names a Collection's URL after the Collection unless you change it. If your useful content lives in a Collection called "Insights" or "Articles", it will be skipped. Tick Also learn blog / news posts when you start the crawl, or enter the blog address itself, which counts as asking for posts.

Our guide to which pages a chatbot should learn from goes through the same choice for static pages.

Check your Collection template's markup

The crawler turns each page's HTML into plain text. A few details of that process matter for CMS templates:

• It drops page furniture. Text inside header, nav, footer and aside elements is removed, along with scripts. That keeps every page's menu and footer from repeating in the knowledge base. If your template places a key field, such as the price or the area served, inside an element that renders as an aside or header, that field is lost. Check the published page source.
• It reads the HTML your server sends. Content that a script fills in after the page loads isn't seen. Webflow renders CMS fields into the page itself, so native Collection content is fine.
• Thin pages are skipped. A page with only a sentence or two of text, such as a team card with a name and a job title, is too short to be stored.
• Long pages are trimmed. Roughly the first 24,000 characters of text on each page are kept.

The page title is kept with the text, so a clear item name helps the assistant match questions to the right item.

Publish, then re-crawl

Webflow's help center says custom code and design changes appear in preview but don't go live until you publish, and the same is true for new CMS items. Two practical rules follow.

Crawl the live domain. Webflow lets you publish to the webflow.io staging subdomain without updating your custom domain. If you've only published to staging, your live site and sitemap don't have the new items yet.

Re-crawl after each publish. Re-crawling is available on Starter and above. When a page is crawled again, its old knowledge is replaced with the new text, so an edited price doesn't leave the old price behind. The Free plan includes a one-time setup from up to 10 pages for the first chatbot, without re-crawls.

One gap to cover by hand: a re-crawl updates the pages it finds, but it doesn't delete knowledge from items you've since unpublished. The knowledge base groups entries by page path, so find the old item's path and use Delete all on it.

Big Collections and the 150-page cap

Each crawl reads up to 150 pages. A Collection with hundreds of items can use that up before your service and policy pages are reached. Two options:

• Point the crawl at your main pages and add the Collection separately.
• Upload the Collection as a CSV. Webflow's help center says you can export a Collection, or selected items, to CSV from the CMS panel. SpideyChat accepts CSV uploads and turns each row into "column: value" lines. Before uploading, delete columns that add noise (image URLs, IDs) and remove archived items, which Webflow's export includes. Rich text fields export as HTML, so a plain text column is cleaner if you have one.

An uploaded file doesn't refresh itself, so re-export and re-upload when the Collection changes.

What to do after the first crawl

Ask the assistant five questions that only your CMS pages answer, then check the unanswered questions in your dashboard after a week. Gaps usually point to a Collection field that isn't on the template, or a Collection that was skipped. The train your bot doc covers documents and Q&A pairs, and the Webflow integration page has the install steps and a free test you can run against your published site.