Knowledge Index Freshness Guide

How the knowledge index works

Your bot does not search the internet or your website in real time during a conversation. It searches a pre-built index — a snapshot of your content converted into vector embeddings. The bot is only as current as the last time the index was updated.

The index is built from multiple sources depending on your setup:

• Your website — pages crawled by the Velaro web scraper
• Platform catalog — products pulled from BigCommerce, Shopify, WooCommerce, Magento, or Square
• KB articles — your admin-authored knowledge base articles
• Uploaded documents — CSV/Excel price lists, PDFs, product sheets

How often does the index update?

KB articles: Indexed immediately every time you publish or edit an article. The daily auto-reindex is a safety net that catches any articles that changed without triggering the live push. If nothing changed, this run costs nothing.

Platform products (BigCommerce, Shopify, etc.): Synced based on your plan's reindex frequency. Only products that changed since the last sync are re-embedded — unchanged products are free to process.

Web scraper (multiple URL sets): Each URL set (a group of starting URLs + crawl settings) has its own independent schedule. You can have multiple URL sets — for example one for your product pages, one for your blog, one for your support documentation. Each crawls on the schedule set by your plan. Only pages that changed since the last crawl are re-indexed; unchanged pages cost nothing.

Can I have multiple website sections crawled separately?

Yes. You can create multiple URL sets under Knowledge → Website. Each URL set:

• Starts from its own seed URLs
• Crawls to a configurable depth
• Runs on its own auto-rebuild schedule
• Tracks its own "last run" and "pages changed" metrics

This means your products section, blog, and help docs can each have their own crawl frequency if needed.

What does "re-index" actually mean?

Re-indexing does not re-process everything every time. Velaro uses SHA-256 content hashing:

1. Crawl the page or fetch the product/article

2. Compute a SHA-256 hash of the content

3. Compare against the stored hash from the last run

4. If the hash matches → skip (zero embedding cost)

5. If the hash changed → generate a new embedding and update the index

At a 1% daily change rate on a 10,000-product catalog, you'd re-embed roughly 100 products per day — about $0.01/day in embedding costs.

How often does re-indexing happen by plan?

The "Large Catalog" add-on also raises your product limit to 500,000 products indexed.

Can I trigger a reindex manually?

Yes. In Knowledge → Index Freshness, click Reindex Now (KB articles) or Run Now (web scraper URL sets). Manual KB reindexes are always delta-safe — they only re-embed articles that changed. Manual scraper runs count against your monthly page crawl quota.

What happens to my index if I cancel?

Your knowledge index is retained for 30 days after cancellation, giving you a window to reactivate and pick up exactly where you left off. After 30 days, the index is cleared automatically. Your actual content (KB articles, conversations, contacts) follows the standard 90-day data retention policy.

Resilience — what happens if a scrape is interrupted

Long crawls are checkpointed page-by-page as they run. If the scraper restarts mid-job for any reason, the next attempt resumes from the last completed page rather than starting over — pages already ingested are not re-fetched or re-billed. The job stays "Running" in the console and finishes normally.

Combined with smart scan, this means scheduled rebuilds are reliable and predictable: unchanged content is detected and skipped, interrupted runs resume cleanly, and each URL set / document folder / YouTube channel / SharePoint scope runs on its own independent schedule without affecting the others.

Is price/availability ever out of date?

No. Price, inventory, and stock status are never stored in the index — they are fetched live from your store (Shopify, BigCommerce, WooCommerce, or Magento) at the moment the bot answers. Reindexing only affects product title, description, tags, and variant text used to help the bot find the right product — it has no effect on what price or stock level the bot quotes, because those are never cached in the first place.

This means you never need to reindex just because a price changed. If a customer asks for a price and the bot answers, that number came from your store's live catalog seconds earlier — not from a snapshot.

Why are product comparisons and searches fast even with a huge catalog?

The bot never scans your entire catalog to answer a question. Every product lookup — search, comparison, or quote — runs a targeted, capped query:

• A single-product lookup or search calls your store's live API directly for just the matching items.
• A comparison checks a small, fixed number of products at once (never more than a handful), not your whole catalog.
• Indexed search (used for "find me something like X" style questions) queries a pre-built search index, not a table scan — response time doesn't grow with catalog size.

This is why a store with 100 products and a store with 100,000 products both get sub-second answers.

What if the bot is giving outdated answers?

If the bot doesn't know about a new product, or its description/tags are stale, it means your index is behind (this never applies to price/availability — see above). Steps to fix:

1. Check Knowledge → Index Freshness — look at "Last Updated" per source

2. If it's been longer than your plan interval, click "Reindex Now" / "Run Now"

3. For platform products — confirm the integration is connected (Integrations → your platform → Test Connection)

4. For KB articles — confirm the article is published and "Include in bot responses" is enabled

5. For scraped pages — confirm the URL set includes the correct starting URLs and crawl depth covers the changed pages