Training Your AI Bot with Web Scraping

What is Web Training?

Web Training lets your Velaro AI bot learn directly from your own website. You provide one or more URLs and the scraper visits them, reads the content, and indexes it so your bot can answer customer questions using your actual site information — pricing, FAQs, product details, policies, and more.

Important: The scraper is designed exclusively to train on your own content. Velaro's terms of service require that you only point the scraper at websites you own or have explicit permission to index. It is not a general-purpose web scraper.

How the Scraper Works

The scraper uses a real browser (not a simple HTTP request) so it can handle JavaScript-rendered pages, dynamic content, and modern websites. It visits each URL, extracts the text content, and stores it in your AI knowledge index.

When a scrape job completes, you click Publish to Index to make that content available to your bot. From that point on, your bot can cite that knowledge when answering questions.

Site Protection and Access

Many websites use security services to prevent automated access. These tools protect against malicious bots but can also block legitimate training runs. The scraper uses several techniques to work around common blocks — but some sites have protections that require action from you.

What the scraper already tries automatically

When the scraper encounters a block, it attempts up to five bypass strategies in sequence before reporting the page as blocked:

1. Browser warmup + wait — visits the site homepage first to establish a real session, then waits 12 seconds for JavaScript security checks to resolve on their own

2. Chrome TLS impersonation — sends the exact network fingerprint that real Chrome sends, bypassing checks that flag "bot-like" network behavior

3. Firefox stealth mode — switches to a patched Firefox browser, which has a completely different fingerprint from Chromium — many sites that block headless Chrome pass Firefox without issue

4. FlareSolverr challenge solver — if enabled on your account, uses a dedicated challenge-solving browser to complete Cloudflare's JavaScript check

5. Stealth HTTP fallback — attempts a direct HTTP request with realistic browser headers as a final option

The scraper tries all of these before marking a page as blocked. Customers do not need to configure any of this — it happens automatically.

What the scraper cannot bypass

Some sites use Cloudflare Turnstile — the interactive "I'm not a robot" checkbox — or similar CAPTCHA systems. These require a human to solve and cannot be completed automatically. This is an intentional security measure designed to stop all bots, including legitimate ones.

If your site uses this level of protection, you have two practical options:

Option 1 — Allow the Scraper by IP (Recommended)

This is the cleanest solution. You temporarily or permanently allow the scraper's IP address in your security settings:

1. Log in to your Cloudflare dashboard (or your site's firewall panel)

2. Go to Security → WAF → IP Access Rules (Cloudflare) or your firewall's equivalent

3. Add the scraper's IP address with action = Allow

4. Contact Velaro support to get the current scraper IP address

For Cloudflare Bot Fight Mode, you can also go to Security → Bots and temporarily set Bot Fight Mode to Off, run the scrape, then turn it back on.

Your IT team or the person who manages your domain/hosting can do this in minutes.

Option 2 — Provide Your Content Directly

If you cannot modify your site's security settings, you can upload your content directly. This is just as effective for training your bot — sometimes more so, because you control exactly what the bot learns.

Supported upload formats:

• PDF — product manuals, policy documents, price sheets, guides
• Word documents (.docx) — internal documentation, SOPs, scripts
• Excel/CSV — product catalogs, FAQs in spreadsheet form, pricing tables
• Plain text or Markdown — any text content
• Images (.jpg, .png) — the scraper reads text from images using OCR
• JSON — structured data you want the bot to use directly

How to upload: Go to AI → Knowledge → File Training and drag your files in. The bot processes them immediately without needing to scrape your website at all.

Copy-paste option: If the content is short (a FAQ page, a policy, a product description), you can also create it as a Knowledge Base article directly in Velaro and enable it for bot use. Go to Knowledge Base → Articles → New Article.

Why Some Sites Block Scrapers

Sites use bot protection for legitimate reasons: to prevent competitors from copying their content, reduce server load, and protect against malicious crawlers. When you are training your own bot on your own content, you are the authorized owner — but the security system has no way to know that without you explicitly granting access.

This is why the "allow by IP" approach works: you are telling your own security system "this specific visitor is me."

Tips for Better Scrape Results

• Include direct links to key pages rather than just the homepage — the scraper follows links to a configured depth, but starting from your FAQ page, pricing page, or product catalog gets you the most relevant content faster
• Run the scrape again after making content changes on your site — the bot only knows what was indexed at the time of the last successful scrape
• Use Smart Scan (the toggle in the job settings) for re-runs — it skips pages that haven't changed and only re-indexes updated content
• Combine sources — many customers scrape their public website AND upload internal PDFs (staff training guides, expanded FAQs) for the best bot knowledge coverage

Activity Log

Every scrape job shows an Activity Log button. Click it to see exactly what happened to each URL — whether it succeeded, was blocked, what technique succeeded, and specific guidance if a block occurred. If the scraper tried all five bypass strategies and still couldn't access a page, the activity log will say so and explain why.

Getting Help

If the scraper cannot reach your site after trying all techniques and your IT team cannot modify the security settings, contact Velaro support. In many cases, we can help you identify the exact block type and the quickest path to get your content indexed.