Crawl a website

A website source is a second way to feed a collection, alongside the S3-compatible buckets Connect a source covers. Give it one or more starting pages and the platform discovers the rest on its own — no bucket, no file list, just a URL.

Everything Connect a source says about schedules, Sync now, the run log, and what happens when a document is edited, removed, or deleted while its source still has it applies to a website source unchanged. This page covers what is specific to crawling: how pages are discovered, which provider actually fetches them, and the rules that keep the crawl from reaching somewhere it shouldn’t.

Add a website source

Open a collection’s Sources tab, choose Add source, and pick Website. The one required field is:

• Seed URLs — one or more starting pages. Everything else is discovered from these.

Everything past that has a sensible default, and most crawls need to touch nothing else:

• Scope — which discovered links get crawled: the same domain as a seed (the default), only under a seed’s own path (a seed of /docs/install scopes to /docs/), or the seeds themselves with no crawling at all.
• Max depth — how many hops from a seed the crawl follows outbound links. 0 crawls only the seeds.
• Max pages — a ceiling on how many pages one run discovers.
• Use sitemap — prefer the site’s sitemap over following links, when one exists. On by default; see below.

Under Advanced: whether robots.txt is respected, a requests-per-second rate limit, include/exclude URL patterns, follow-only patterns for pages to crawl through without storing, whether linked documents (PDFs and the like) are followed, which fetcher renders the page, and the credential a non-default fetcher needs. All covered below.

Test and preview before you save

Test connection runs a real check — the first seed’s robots.txt, whether it’s reachable, and how many URLs its sitemap lists — the same first call an actual sync makes, not a bare reachability ping.

Preview goes a step further: it drains the crawl’s listing, up to a bound on both page count and wait time, and shows you the URLs it would ingest — without ingesting a single one. Nothing is written: no source, no document, no item. Use it to sanity-check scope and patterns before committing to a schedule.

Sitemap vs. link crawl, and when each is right

A website source discovers pages one of two ways, and the difference matters for how expensive an unchanged site is to keep in sync.

Sitemap mode reads the site’s sitemap.xml (found via robots.txt’s Sitemap: lines or the conventional /sitemap.xml path) and never fetches a single page body to build its listing — it only ever reads robots.txt and the sitemap documents themselves. Each URL’s <lastmod> becomes its version, so an unchanged page is skipped without being downloaded at all. This is the cheap, preferred path, and it’s why Use sitemap defaults to on.

Link crawl is the fallback: a bounded walk outward from your seeds, following links up to Max depth. Here, discovery is fetching — a page’s outbound links aren’t knowable without reading its body — so every in-scope page is genuinely re-fetched on every run. That is inherent to crawling a site with no sitemap, not a shortcut this platform is missing. If a site you crawl often is large, ask whoever runs it for a sitemap rather than leaning on link crawl indefinitely.

The choice is made separately for each seed. If you configure two seeds and only one of them has a sitemap, that seed is synced from its sitemap while the other is still link-crawled — a source is never forced into all-sitemap or all-link-crawl by whichever seed happens to answer first.

Hitting Max pages marks the listing partial, and a partial listing prunes nothing: nothing already ingested is treated as removed on a run that stopped short of seeing the whole site. The same applies if a seed itself is unreachable, blocked by robots.txt, or excluded by your own scope or pattern settings — the crawl never reached its own starting point, so nothing it saw can be a complete picture of the site.

Choosing a fetcher

Fetcher decides how one page’s HTML is obtained. It does not change anything else — discovery, scope, robots.txt, rate limiting, change detection, and pruning are the platform’s own regardless of which fetcher you pick.

Fetcher	Mechanism	Credential	When to use it
Auto (default)	A plain fetch, and the built-in browser for pages whose content is not in their HTML	None	Leave it here unless you have a reason not to.
Direct	A plain fetch, never rendered	None	A site you know is server-rendered, when you want the fastest possible crawl.
Rendered	The built-in browser, on every page	None	A site where every page needs JavaScript and you would rather not pay for the detection.
Jina Reader	Renders the page via r.jina.ai	Optional (raises a rate limit)	A site that blocks our own browser.
Firecrawl	Renders the page via the Firecrawl API	Required	Same as Jina; Firecrawl base URL lets you point at a self-hosted Firecrawl instead of the public API.

How Auto decides

Auto fetches every page normally first and looks at what came back. Unresolved template placeholders (stray {{ ... }} still in the text), an empty application container, or a page left with almost no readable text after its navigation and boilerplate are removed — any of these means the real content is being drawn in by JavaScript, and that one page is fetched again through the built-in browser. Auto also watches the crawl as a whole: if several different pages come back with identical body text, something is wrong with the fetch rather than with the site, and it starts rendering the rest of the crawl too.

If your deployment has no built-in renderer available, Auto behaves exactly like Direct — and says so the next time you test the connection, rather than leaving you with a collection full of navigation menus and no explanation. Switch Fetcher to Jina Reader or Firecrawl if you need rendering and no built-in renderer is available.

None of the rendering options are given the run of the site: the built-in browser and both providers only ever render one page at a time, never crawl on their own. The platform’s own discovery, scoping, and change detection still apply on top of whatever comes back — a fetcher changes how a page’s markup is obtained, never how the site is crawled.

An API key is write-only, same as any other source credential: once one is stored, the field just says the value is set, and leaving it blank on a later edit keeps it unchanged.

What a crawled page becomes

Each page becomes one document. Its navigation, header, footer and side rails are removed; its headings become sections, so a collection that chunks by Structure gets one chunk per section. The page’s <h1> (or its <title>) is the document’s title; an <h1> hidden on the page, such as one inside a pop-up that never opens, does not count. If your collection still chunks by Fixed size, switch it to Structure in the collection settings — the Sources dialog reminds you when that applies.

robots.txt and rate limiting

robots.txt is honoured by default — a path it disallows is never fetched, whether that page would have come from the sitemap or a link crawl. Turn Respect robots.txt off only for a site you actually own, such as a staging environment that disallows every crawler by default.

The crawl rate-limits itself per host at Requests per second. If the site’s own robots.txt asks for a slower Crawl-delay, that raises the effective rate limit — it never lowers it below what you configured.

Linked documents

A link to a PDF or another format the platform can ingest (matched by its URL’s file extension) is fetched once as its own item and never crawled further for links of its own — a document is always a leaf, not a page the crawl continues from. This is on by default (Follow linked documents) because a help centre whose real content lives in linked PDFs is common enough that leaving it off by default would make the source look broken. Turn it off if you want the site’s own pages only.

What gets blocked, and why

The crawler will only ever talk to public addresses over http or https. Before connecting anywhere, it resolves the host and checks every address that name resolves to — not just the one it happens to use — refusing anything private, loopback, link-local, multicast, reserved, or the cloud metadata endpoint (169.254.169.254). Every redirect is followed by hand and re-checked the same way, so a seed that passes but redirects somewhere internal is caught at the redirect, not let through.

This check runs wherever the connector actually tries to reach a URL — testing the connection, previewing, or a scheduled sync — not the moment you click Save. Saving a source only validates that your seed URLs are well-formed; it does not resolve DNS or open a connection. So you can save a source pointed at http://localhost/ or an internal hostname without an immediate error, and the platform will refuse it the first time it actually tries to reach it — as a failed connection test, a preview that reports nothing but a note explaining why, or a failed sync run. If you point a seed at an internal host and wonder why nothing happens, this is why: it isn’t a crash, it’s a page the platform will never connect to. Test connection before saving is the fast way to find out.

A page over the byte cap (5 MB) is truncated, not refused — a large documentation page is still mostly useful content once cut down to size, unlike an oversized binary, which is a different failure mode entirely.

Getting cleaner pages

Most sites mark their own furniture — navigation, headers, footers, cookie dialogs — and it is removed for you. Some mark none of it, and then every page arrives wrapped in the site’s menu and its cookie notice, which ends up in your search results.

Content selector fixes that. It is a CSS selector naming the part of each page that holds the real content, and everything outside it is ignored. Open your browser’s inspector on one article, find the element that wraps the body, and use its class or id:

what you see in the inspector	what to type
<div class="article-body">	.article-body
<div id="content">	#content
<main>	main

One selector usually works for a whole site, because pages share a template. If it matches nothing on a page, that page is used whole and the run log says so, rather than being ingested empty. The same happens when the selector does match but everything under it turns out to be furniture — a site that reuses one class name for both its articles and its cookie dialog would otherwise give you fifty identical documents of cookie policy.

So a selector can only ever improve a page or leave it alone, and the run log tells you which happened on every page. If you see matched no real content against most of a run, the selector is wrong for that site: open a page the crawl actually visited, rather than the one you picked the selector from, and check what wraps its body there.

Leave it blank unless you need it. Automatic cleanup handles most sites.

Shops and other sites reached through listing pages

An online shop without a sitemap is usually reached only through its category pages: the home page links to a few categories, and each category lists the products. Excluding the categories stops the crawl at the home page, because an excluded page is never fetched and its links are never found. Keeping them stores every category as a document too — near-identical grids of product names and prices that crowd your search results.

Follow-only patterns solve this. A page whose URL matches one is crawled for its links but never stored, so the crawl reaches every product without keeping the listings. For a typical shop that is the category path, for example /category/. Follow-only pages have their own limit, the same number as Max pages, so a very large category tree cannot run on forever; reaching it marks the listing partial, exactly as Max pages does. In sitemap mode a matching URL is simply skipped, since the sitemap already lists the pages it would have led to.

Sections: parts of a site on their own schedules

A news page changes every hour; an archive changes once a year. Instead of crawling the whole site as often as its busiest part, split the source into sections. Each section has a name, one or more patterns, its own schedule, and optionally its own start URLs:

Section	Pattern	Start URL	Schedule
News	/news/	https://example.com/news/	Hourly
Everything else	—	the source’s seed URLs	Weekly

A page belongs to the first section, from the top, whose pattern matches its address; anything no section matches belongs to Everything else, which follows the source’s own schedule. Patterns are regular expressions searched anywhere in the address, as for include and exclude patterns. Test an address in the dialog shows which section a page lands in before you save.

A section’s sync crawls only inside its own patterns, from its own start URLs when it has them, and adds and updates its own pages. A page removed from the site is noticed by the source’s own sync (Everything else), which looks at the whole site. The source’s own sync also starts from every section’s start URLs, so it sees the whole site. Scope, robots.txt, rate limits and page limits stay the source’s, and a section’s start URLs must lie inside the pages the source crawls. A source runs one sync at a time; when several parts are due, the one that has waited longest goes first. Each section’s syncs appear in the source’s sync log, where you can also sync one section now. The collection’s freshness check judges each section on its own schedule.

S3 buckets can have sections too, matched against the object key. Other source types cannot have sections yet.

Watching a crawl

The source row shows progress while a sync runs. When the site has a sitemap the bar is a real proportion, because the number of pages is known before anything is fetched. A link crawl has no total to measure against — its list of pages grows as it reads — so it shows a counter instead.

The most recent run also carries its own log, alongside the run history Connect a source already covers. It updates while the run is live and settles once the run finishes. It records what’s worth reading rather than every page:

• where the pages came from — a sitemap and how many it held, or a link crawl from the seed and how far it was allowed to go
• a seed that was refused, and why, so a mistyped or internal address reads as a refusal rather than a site with no pages
• a page whose content turned out not to be in its HTML when no renderer is available, which is your signal to change the fetcher
• any page that could not be fetched, or has disappeared from the site
• when the crawl had to stop early, and what was removed after the grace period
• the closing line, including how many pages needed a browser to render

Pages that fetch normally are already counted in the run’s own outcome numbers, so they aren’t listed again here. Tick Warnings and errors only to see just the problems.

Next: Bulk import.