How the knowledge index works

Your bot does not search the internet or your website in real time during a conversation. It searches a pre-built index — a snapshot of your content converted into vector embeddings. The bot is only as current as the last time the index was updated.

The index is built from multiple sources depending on your setup:

• Your website — pages crawled by the Velaro web scraper
• Platform catalog — products pulled from BigCommerce, Shopify, WooCommerce, Magento, or Square
• KB articles — your admin-authored knowledge base articles
• Uploaded documents — CSV/Excel price lists, PDFs, product sheets

How often does the index update?

KB articles: Indexed immediately every time you publish or edit an article. The daily auto-reindex is a safety net that catches any articles that changed without triggering the live push. If nothing changed, this run costs nothing.

Platform products (Shopify, BigCommerce, WooCommerce, Magento): Synced on a schedule, and on demand for Shopify and BigCommerce (see "How product sync works" below). Unchanged products are skipped, and a price, stock or link change updates the existing entry without re-embedding.

Web scraper (multiple URL sets): Each URL set (a group of starting URLs + crawl settings) has its own independent schedule. You can have multiple URL sets — for example one for your product pages, one for your blog, one for your support documentation. Each crawls on the schedule set by your plan. Only pages that changed since the last crawl are re-indexed; unchanged pages cost nothing.

Can I have multiple website sections crawled separately?

Yes. You can create multiple URL sets under Knowledge → Website. Each URL set:

• Starts from its own seed URLs
• Crawls to a configurable depth
• Runs on its own auto-rebuild schedule
• Tracks its own "last run" and "pages changed" metrics

This means your products section, blog, and help docs can each have their own crawl frequency if needed.

What does "re-index" actually mean?

Re-indexing does not re-process everything every time. Velaro uses SHA-256 content hashing:

1. Crawl the page or fetch the product/article

2. Compute a SHA-256 hash of the content

3. Compare against the stored hash from the last run

4. If the hash matches → skip (zero embedding cost)

5. If the hash changed → generate a new embedding and update the index

Only content whose text really changed is re-embedded, so a typical day touches a small fraction of a catalog.

For KB articles the check covers the title and the text. Saving an article without changing either does no indexing work. Changing the title or the text reindexes that article.

How product sync works (Shopify, BigCommerce, WooCommerce, Magento)

What starts a sync:

• A scheduled sync that runs on its own for every connected store.
• An on-demand sync for Shopify and BigCommerce: Sync Now on the platform's products page in Velaro, or velaro index reingest --source shopify (or bigcommerce) in the CLI.
• WooCommerce and Magento sync on their schedule only. There is no on-demand button for them yet.

What each sync checks, per product: two fingerprints are kept for every product.

1. The searchable text (title, description, tags, variants). If it changed, the product is re-embedded.

2. The structured details: lowest and highest price, in-stock flag, brand, category, product link and the active flag. If only these changed, the existing index entry is updated in place with no re-embedding, so the new price or stock level reaches the bot's search results on the next sync.

3. If neither changed, nothing is written.

The first sync after this behaviour went live refreshes the structured details once for products that were already indexed, then skips them as usual.

Quick versus slow: price, stock, brand, category, link and active-flag changes are quick (an in-place update). Description or title changes take a re-embed, which is a little slower but still automatic.

How to tell a product is stale:

• Ask Moshky or run product_sync_status (MCP) to see the last successful sync time, the number of products indexed and whether any indexing errors are outstanding.
• Check Knowledge → Index Freshness, "Last Updated" for the product source.
• Compare the product in your store with what the bot says. A product that was just edited in your store stays as it was until the next sync.

How to force a resync: Shopify and BigCommerce, use Sync Now (or velaro index reingest --source shopify|bigcommerce). WooCommerce and Magento, wait for the next scheduled sync. If a product is still wrong after a sync, check that the integration is connected (Integrations → your platform → Test Connection).

Daily indexing safety cap

To protect every account from runaway re-indexing, each site has a daily cap on how much new content can be embedded, covering KB articles, product catalogs, conversation training, uploads and web crawls. Normal use never reaches it, and unchanged content never counts toward it.

If a site reaches the cap, indexing for that site pauses for the rest of the day and the affected items resume automatically the next day. They are not lost. What you will see: new or edited items are not searchable by the bot until the next day, and the event is logged for Velaro staff. Limits depend on your plan; see your plan under Billing, Usage Overview. If it keeps happening, open a support ticket from the Help menu or ask Moshky so staff can review the site.

How often does re-indexing happen by plan?

Higher plans re-index web and product content more often, and the Large Catalog add-on re-indexes daily. KB articles are also re-checked daily as a safety net on every plan. An admin or billing user can see their plan's schedule on Billing.

The "Large Catalog" add-on also raises your product limit; the exact number is on Billing.

Can I trigger a reindex manually?

Yes. In Knowledge → Index Freshness, click Reindex Now (KB articles) or Run Now (web scraper URL sets). Manual KB reindexes are always delta-safe — they only re-embed articles that changed. Manual scraper runs count against your monthly page crawl quota.

What happens to my index if I cancel?

Your knowledge index is retained for 30 days after cancellation, giving you a window to reactivate and pick up exactly where you left off. After 30 days, the index is cleared automatically. Your actual content (KB articles, conversations, contacts) follows the standard 90-day data retention policy.

Resilience — what happens if a scrape is interrupted

Long crawls are checkpointed page-by-page as they run. If the scraper restarts mid-job for any reason, the next attempt resumes from the last completed page rather than starting over — pages already ingested are not re-fetched or re-billed. The job stays "Running" in the console and finishes normally.

Combined with smart scan, this means scheduled rebuilds are reliable and predictable: unchanged content is detected and skipped, interrupted runs resume cleanly, and each URL set / document folder / YouTube channel / SharePoint scope runs on its own independent schedule without affecting the others.

Is price/availability ever out of date?

When the bot looks up a specific product, it asks your store (Shopify, BigCommerce, WooCommerce or Magento) at that moment, so the price and stock it quotes come from your live catalog.

Index entries used for search ("find me something like X") also carry price, stock, brand, category, link and active flag. Those are refreshed on every product sync without re-embedding, so search results can be behind your store by up to one sync cycle. If you need them current right away, run Sync Now (Shopify and BigCommerce).

Why are product comparisons and searches fast even with a huge catalog?

The bot never scans your entire catalog to answer a question. Every product lookup — search, comparison, or quote — runs a targeted, capped query:

• A single-product lookup or search calls your store's live API directly for just the matching items.
• A comparison checks a small, fixed number of products at once (never more than a handful), not your whole catalog.
• Indexed search (used for "find me something like X" style questions) queries a pre-built search index, not a table scan — response time doesn't grow with catalog size.

This is why a store with 100 products and a store with 100,000 products both get sub-second answers.

What if the bot is giving outdated answers?

If the bot doesn't know about a new product, or its description, tags, price or stock in search results are stale, your index is behind (a specific-product lookup is always live, see above). Steps to fix:

1. Check Knowledge → Index Freshness — look at "Last Updated" per source

2. If it's been longer than your plan interval, click "Reindex Now" / "Run Now"

3. For platform products — confirm the integration is connected (Integrations → your platform → Test Connection)

4. For KB articles — confirm the article is published and "Include in bot responses" is enabled

5. For scraped pages — confirm the URL set includes the correct starting URLs and crawl depth covers the changed pages