Knowledge

Add the sources the agent answers from, accept the ones your coding agent proposes, train them, keep them current, and see when ARCYBot follows robots.txt.

Add and train the sources the agent answers from on Configure > Knowledge.

Before you start

A Web crawler needs a Site under a verified Domain (Domains), except one 50-page read of the Workspace creator's work email domain (that name or its www).
A Walkthrough always needs one.
Your Plan caps how many sources you hold (Billing).

Setting it up

Add source offers five types:

Web crawler. Give it a Starting URL. It follows your sitemap and links below that path. Turn off Allow linking to this source in answers to keep pages unlinked. For Follow robots.txt, see ARCYBot.
Flows. Pick Flows to suggest (Flows).
Text. 15,000 characters at most, Markdown included. What every answer needs belongs in Persona.
File upload. PDF, DOC, DOCX, TXT or MD, 10 MB at most. To replace one, delete the source and add it again.
Walkthrough. Pick a Site and path under Starting page, then Start walkthrough. Your Product opens in a new tab, and each screen you visit becomes one document of its controls and fields. A dialog or tab you open is a screen of its own; a menu joins its screen. Never captured: what you type, table rows, select options, on or off states, screenshots. One per Site. Stop saves and leaves the tab open; the row reads Captured.

Hand it to your agent with the ARCY MCP opens the Add training sources entry on ARCY MCP. Each proposed source shows here as Proposed by your agent; Accept adds and trains it, Discard drops it.

Press Train on the row. Runs are under Training log, pages under Documents. The row itself says how far a running crawl has read.

A second Train reads only new pages. To reread changed pages, use Fetch this document again on one page, or Retrain from the start, which deletes every saved page and crawls again. Leave out of future crawls drops one page for good. Stop training keeps what it saved.

Walk again replaces every screen on Stop. Leave this screen out of training survives a re-walk.

What it changes

One crawl saves 1,000 pages; a larger site continues on the next Train.
A page over 5 MB is skipped and named in the log.
One Web crawler per Site, and a crawl stays on it.
Each Train drops saved pages robots.txt now closes, and rereads them once reopened.
Crawled and walked pages cannot be edited: leave one out and add a Text source instead.
A walk holds 300 screens and stops after 30 minutes.
Training spends no AI credits, whatever the source.
Deleting a source removes its content at once.

ARCYBot

The crawler is ARCYBot/1.0. On a Site under a Domain you proved, or your work email's domain, it sets aside robots.txt rules for every bot (User-agent: *) unless the source has Follow robots.txt on. It always obeys a group naming ARCYBot and waits the Crawl-delay asked for. To keep it out:

User-agent: ARCYBot
Disallow: /

When it goes wrong

An address is refused as not under a verified Domain, or as already read once with your work email. Verify its Domain on Domains.
A crawl fails. Point at Failed for why (more).
Far fewer pages than your site has. Pages are unlinked and there is no sitemap. Publish one.
The walkthrough tab opens without the ARCY bar. The link expired, the Token is another environment's, or arcy.js is missing.