Sources

A Source is one item of material the agent is trained on. It's the only grounding Chat has beyond what it can see on the current page, so this is the primary way you tell it about your product.

Adding a source

From your app's dashboard, open Agent > Train. Four source types are supported:

Web crawler. Give it a starting URL; it finds the rest of the pages from there.
Flows. An authored Flow can itself be a source, so Chat can describe what a Flow does.
Text. Paste content directly.
File upload. PDF, DOC, DOCX, TXT, or MD, 10MB max per file.

Adding a source doesn't make it available to Chat immediately. Press Train on the source to index its content. Training runs in the background and has two halves: the pages are crawled first (Training (12/88)), then every page found is indexed (Indexing (12/88)). The source only reads Trained once the second half is done, because a page is not usable by Chat until it has been indexed. On a large site that half takes a few seconds longer than the crawl. Every training run is logged on the Train > Log page.

How the crawler finds pages

Two ways, and it uses both:

Your sitemap. It reads robots.txt for a Sitemap: line, then tries /sitemap.xml on the same domain. If the sitemap is an index pointing at more sitemaps, it follows those too.
Links on the pages it fetches. Starting from your URL, it follows every link that goes to the same domain and sits at or below the starting path.

A crawl stops at 2,000 pages. It runs in the background, so you do not have to keep the page open while a large site is indexed.

Training is incremental. When you train a source again, ARCY does not retrain pages it has already indexed. It still fetches them to follow their links, then spends the run on pages it has not seen. If the pages themselves have changed and you want ARCY to read them again, use Retrain from scratch in the source menu: it deletes every indexed page and crawls the site from the start.

If far fewer pages were indexed than your site has, the cause is almost always that the pages are not reachable either way. A site whose navigation is a collapsed menu renders those entries as buttons, not links, so nothing on the page points at them, and if the site also has no sitemap the crawler has nothing left to follow. Publishing a sitemap fixes it, and it is the more reliable of the two paths in any case.

What gets indexed from a crawled page

ARCY keeps the content of a page and drops the parts that repeat on every page: the header, the navigation, the sidebar, the footer, and buttons like search or the theme switcher. What is left is kept as markdown, so the page's headings, lists, tables, code blocks and links survive instead of flattening into one block of text. This is what Chat retrieves from, and headings in particular make a passage much easier to find.

To see it, open Agent > Train, then View documents on a source. Each indexed page is listed with the text ARCY extracted from it.

If a page is missing content or carrying something it should not, use the source's include and exclude CSS selectors to point ARCY at the right part of the page.

Editing an indexed page

Pages found by the web crawler cannot be edited. Each one is a copy of a live URL, so any edit would be replaced the next time the page is fetched. Two controls do survive a re-crawl:

Include and exclude selectors on the source, to change what is taken from every page.
Exclude from future crawls on a single document, to drop one page for good.

If a crawled page is wrong and you want to correct it, exclude the page and add the corrected version as a text or file source.

How Chat uses a source

When answering a question, Chat retrieves the most relevant passages across your trained sources and includes them in its context. If retrieval is unavailable for any reason, Chat still answers from what it knows about the current page, so a Sources problem never breaks Chat outright.

Training on your product's content requires enrichment consent for your workspace. If it hasn't been granted, adding a source fails clearly in the dashboard with a link to grant it, rather than silently doing nothing.

Removing a source

Remove a source from Agent > Train at any time. Chat stops drawing on it immediately.