Train

A source is one body of material the agent answers from. You add and train sources on Autopilot > Train.

Before you start

• A Web crawler or a Walkthrough only runs on a domain verified for this environment. See Origins.
• Your Plan sets how many sources you can hold. Past the cap, adding one is refused until you delete one or change Plan. See Billing.

Setting it up

Add source offers five types:

• Web crawler. Give it a Starting URL. It finds the other pages from your sitemap and from links at or below that path. Turn off Allow linking to this source in answers for pages that should stay unlinked.
• Flows. Pick Flows to suggest, so the agent can describe and offer them. See Flows.
• Text. Up to 15,000 characters. Markdown works.
• File upload. PDF, DOC, DOCX, TXT or MD, 10 MB at most. A file cannot be replaced: delete the source and upload again.
• Walkthrough. Choose a Starting page, then Start walkthrough. Your Product opens in a new tab and every screen you visit becomes one document of its interface text: headings, menus, buttons, field labels. Nothing you type, no dropdown options, no table rows and no screenshots are taken. One per Product.

Then press Train on the row. The row reads Training with a page count, then Trained. Runs are listed under Training log, saved pages under Documents with their extracted text.

Training is incremental: a second Train reads only pages it has not seen. To reread pages you changed, use Fetch this document again on one page, or Retrain from the start on the source, which deletes every saved page and crawls again. Leave out of future crawls drops one page for good. Stop training ends a crawl after its current page and keeps what it saved.

When your Product changes, Walk again replaces every screen the moment you stop. Leave this screen out of training survives a re-walk, so an internal page never becomes an answer.

What it changes

• One crawl saves up to 1,000 pages. A larger site continues from page 1,001 on the next Train.
• A page over 5 MB is skipped and named in the log.
• The crawler identifies itself as ARCYBot/1.0 and obeys robots.txt, including Crawl-delay. On a site that blocks all crawlers, add a group for it:

User-agent: ARCYBot
Allow: /

• From a crawled page ARCY keeps the content and drops the header, navigation, sidebar and footer.
• Crawled pages cannot be edited. To correct one, leave it out and add the corrected text as a Text source.
• A Walkthrough writes a short summary per screen on every Train, and the walk stops on its own after 30 minutes.
• Deleting a source removes its saved content at once.

When it goes wrong

• Nothing is fetched and the message names the domain. The domain is not verified for this environment. Verify it on Origins, then press Train again. Pages already saved stay in place.
• The log says pages were skipped because of robots.txt. Your site disallowed ARCYBot. Add the group above. This is not counted as a failure.
• Far fewer pages than your site has. The pages are not linked and there is no sitemap. Publish a sitemap.
• The walkthrough tab opens without the ARCY bar. The link expired, the page carries another environment's Token, or arcy.js is not installed there. The reason is written on the page.

See also

• Persona
• Pulse
• Reference