Connect a source

Uploading documents by hand works, but it means you are the sync mechanism: every time a file changes, someone has to remember to re-upload it. A source replaces that with a bucket the platform checks on its own schedule — new files are ingested, changed files are re-ingested, and files that disappear are taken out of retrieval without anyone opening the Knowledge screen.

A collection can have any number of sources. Each one lives on the collection’s Sources tab, alongside the Documents tab you already know from Ingest a document and query it — a source is just a second, unattended way of getting documents into the same collection.

Add a source

Open a collection’s Sources tab and choose Add source, then pick a source type. Six are available; each has its own guide for what it needs from you and how it detects change:

Source type	What it syncs	Guide
S3-compatible bucket	Objects under a prefix in Amazon S3 or any S3-compatible store	below
Website	Pages discovered from seeds, a sitemap or a bounded crawl	Crawl a website
SharePoint / OneDrive	A document library or a OneDrive, via Microsoft Graph	Connect SharePoint or OneDrive
Google Drive	A Drive folder read by a service account	Connect Google Drive
Confluence	Pages and attachments of spaces, page trees or labels	Connect Confluence
SQL database	The rows of a SELECT, one document each	Connect a database

Whatever the type, give the source a name (yours to choose; it only labels the row). For an S3-compatible bucket — Amazon S3 itself, MinIO, or anything else that speaks the S3 API — fill in:

• Bucket — required.
• Prefix — optional. Restricts the sync to keys under this path, useful when only one folder of a shared bucket belongs to this collection.
• Endpoint — optional. Leave blank for Amazon S3 itself; set it to reach a self-managed S3-compatible server such as MinIO.
• Region — optional.
• Access key ID — required.
• Secret access key — required, and write-only (see below).
• Public base URL — optional. If your bucket sits behind a browsable URL, set this and citations will link back to the original file instead of just naming it.

Only files whose extension the platform knows how to parse are picked up — the same set the manual upload screen accepts. Anything else (unsupported file types, or keys that are really folders) is skipped silently, not reported as an error.

Metadata next to the objects

A bucket may carry metadata files beside the documents, in the same shapes Bulk import accepts: a report.pdf.metadata.json next to report.pdf (a flat object, or Amazon Bedrock’s metadataAttributes shape), or a folder-level _metadata.json, _metadata.csv or .onyx_metadata.json that applies to every object in that folder and below. More specific files override broader ones key by key. Metadata files are never ingested as documents themselves, and editing one re-ingests the documents it governs on the next run. Each document also gets a path metadata value: its folder, relative to the source’s prefix.

Test before you save

Test connection doesn’t just check that the server answers — it runs the same listing call a real sync would, against the bucket, prefix, and credentials currently in the form. A green result means the first sync will actually find something; a reachable-but-wrong-bucket configuration fails here instead of on the first scheduled run.

The secret is write-only

Once a source has a stored secret, the form never shows it back to you — the field simply reads “Leave blank to keep the stored value.” Typing a new value replaces it; leaving the field empty on save keeps whatever is already stored. There is no way to retrieve a secret you have already saved, by design — only to replace it.

Schedules

A source syncs Manually only (use Sync now) or On a schedule:

How often	Options
Every 15 or 30 minutes	For sources that change often.
Hourly	At a minute past the hour you choose.
Daily	At a time of day, optionally only on some days (weekdays, say).
Weekly	At a time of day on the days you choose.
Monthly	At a time of day on a day from the 1st to the 28th.

Times are in the time zone you pick and stay put across daylight-saving changes. The form lists the next five runs before you save. A run that is still going when its next slot comes is not started twice: the next run is the first slot after the current one finishes.

Pick the slowest schedule that keeps answers current: a page that changes monthly gains nothing from a daily check. The collection’s freshness check follows the schedule, so a weekly source is not reported stale after a day.

Changing the schedule takes effect from the next check; it does not retroactively affect a run already in progress.

Websites and S3 buckets can also be split into sections, parts with their own schedules, so a busy part is checked often and the rest rarely. See Crawl a website.

Sync now

Sync now queues an immediate run regardless of the schedule. While a run is already queued or in progress for that source, the button is disabled — and if you somehow get a second request in anyway, the platform refuses it rather than starting a run on top of another. A source only ever has one run in flight at a time, so nothing is ever double-counted or raced.

Reading the run log

Each source keeps a log of its recent runs, open it from the source’s row. Every entry shows when it ran, what triggered it (the schedule or a manual click), how long it took, and an outcome count broken down as:

• seen — items the listing found in total
• added — new documents ingested
• updated — existing documents re-ingested because the file changed
• skipped — items already up to date, left untouched
• removed — items no longer present at the source, taken out of retrieval
• failed — items that could not be fetched or ingested this run

Skipped is the normal outcome, not a warning sign — on most runs almost nothing has changed, and the platform detects that from the object’s own version marker (or a content hash, if the connector has none) without downloading the file again. A run made up almost entirely of “skipped” means the sync is working exactly as intended.

A run that fails partway still records what it managed before stopping, and one bad item never aborts the rest of the run.

What happens when a file changes

The changed file is re-ingested through the same pipeline a manual upload goes through — parsed, chunked, and re-embedded — and it starts answering queries with its new content on the next search.

One exception: if you have paused that document from the Documents tab, a source sync will not un-pause it. A change at the source updates the document’s content in place but leaves your pause in effect — your own curation always outranks what the source reports, so pausing a document is not something a sync can accidentally undo.

What happens when a file is deleted at the source

The moment a sync notices a file is gone, its document leaves retrieval immediately — it stops answering queries on the very next search, before anything is actually deleted. The document itself, and its file, are only removed for good after a grace period (seven days, by default), so a sync run that turns out to have been pointed at a temporarily-empty listing doesn’t cost you anything irreversible.

A sync only ever acts on a listing it trusts completely. If a listing is cut short — a large bucket that takes more than one run to walk, or a run that hits an error partway through — nothing is treated as deleted that run. A partial view of a bucket looks exactly like a bucket someone emptied, so the platform would rather wait for a complete listing than risk pruning documents that are still there.

Deleting a document that’s still at the source

If you delete a document from the Documents tab while its source still has the underlying file, expect it back: the next sync will see the file is still there and re-add it. When you see a delete confirmation warning about this, it’s telling you exactly that — deleting the document doesn’t stop the source from bringing it back.

Pause the document itself, from the Documents tab, to keep it out of retrieval for good: a paused document stays out of every search, and a sync will never undo the pause — a changed file still updates its content, but leaves the pause in place. Pausing the source instead is a much heavier tool: it stops that source’s schedule and its “Sync now” button entirely, so every document it manages stops being kept up to date, not just the one you meant to hold back. Reach for it only when you actually want the whole source to stop, never as a stand-in for pausing one document.

Next: Crawl a website.