Train

Add and train the sources the agent answers from on Configure > Train.

Before you start

• A Web crawler or a Walkthrough only runs on a verified domain. See Origins.
• Your Plan caps how many sources you hold (Billing).

Setting it up

Add source offers five types:

• Web crawler. Give it a Starting URL. It follows your sitemap and links below that path. Turn off Allow linking to this source in answers to keep pages unlinked.
• Flows. Pick Flows to suggest (Flows).
• Text. 15,000 characters at most, Markdown included. This is the long reference looked up when a question needs it; the short text in every answer is Persona.
• File upload. PDF, DOC, DOCX, TXT or MD, 10 MB at most. To replace a file, delete the source and add it again.
• Walkthrough. Choose a Starting page, then Start walkthrough. Your Product opens in a new tab. Each screen you visit becomes one document: its controls by part of the screen, heading and position, and a field's type, placeholder, required and disabled. A dialog or tab you open is its own screen; an expanded section or opened menu joins its screen. A click that opened a screen is written on both ends, so the agent answers with the path and what opens next. Never captured: what you type, table rows, select options, on or off states, screenshots. One per Product. Stop starts training and closes the tab; if it could not start, the row reads captured and Train starts it.

Hand it to your agent with the ARCY MCP opens the Add training sources entry on ARCY MCP. Each proposed source shows here as Proposed by your agent; Accept adds it and starts training, Discard drops it.

Press Train on the row. Runs are under Training log, pages under Documents.

A second Train reads only new pages. To reread changed pages, use Fetch this document again on one page, or Retrain from the start, which deletes every saved page and crawls again. Leave out of future crawls drops one page for good. Stop training keeps what it saved.

Walk again replaces every screen and path on Stop. Leave this screen out of training survives a re-walk.

What it changes

• One crawl saves 1,000 pages; a larger site continues on the next Train.
• A page over 5 MB is skipped and named in the log.
• The crawler is ARCYBot/1.0 and obeys robots.txt, including Crawl-delay. On a site that blocks every crawler, add:

User-agent: ARCYBot
Allow: /

• A crawl keeps the content and drops the navigation. Text repeated across the site is saved once, in a document named after the site.
• One Web crawler per site: a second source on the same domain is refused.
• Crawled and walked pages cannot be edited. To correct one, leave it out and add a Text source.
• A walk holds 300 screens, dialogs included, and stops after 30 minutes. Each screen gets a summary.
• Deleting a source removes its content at once.

When it goes wrong

• Nothing is fetched and the message names the domain. Verify it on Origins and press Train again.
• The log says pages were skipped because of robots.txt. Add the ARCYBot group above (more).
• Far fewer pages than your site has. Pages are unlinked and there is no sitemap; publish one.
• The walkthrough tab opens without the ARCY bar. The link expired, the page carries another environment's Token, or arcy.js is missing; the page says which.

See also

• Persona
• Conversations
• Reference