Bulk import

Uploading one file at a time works until you have hundreds of them. Bulk import moves a whole folder — from your disk, from a migration export, or from a script — into a collection in one operation, with one report at the end naming exactly what happened to every file.

You need a key with the editor role.

export KOVAC_API_KEY="the-key-you-created"
export KOVAC_BASE="https://your-platform-host"
export COLLECTION_ID="the-collection-you-are-importing-into"

Three ways in

Way	Best for	How metadata arrives
Zip archive	A folder you already have on disk	Sidecar files inside the zip
Folder upload	The browser’s own folder picker, or a script that already knows each file’s metadata	Passed per file on each request
Batch manifest	A script pulling files from elsewhere — URLs, or a mix of URLs and local files	Passed per entry in the manifest

All three land in the same place: a collection’s document list, each file going through the same parser, chunker and embedder a single-file upload uses. What differs is only how the bytes and the metadata travel.

1. Zip archive

export IMPORT_ID=$(curl -s -X POST "$KOVAC_BASE/api/manage/collections/$COLLECTION_ID/imports" \
  -H "Authorization: Bearer $KOVAC_API_KEY" \
  -F "archive=@handbook-export.zip" | jq -r '.import_id')

The response is 202 with {"import_id": "...", "kind": "archive"}. The zip is validated immediately — a corrupt archive, an unsafe entry name (../), or one over the file-count ceiling is refused with 400 in this same response, before anything is scheduled. Past that point, extraction and staging run in the background: poll for completion the same way you would for a single document (see Ingest a document and query it), just against the import instead of a document:

curl -s "$KOVAC_BASE/api/manage/imports/$IMPORT_ID" \
  -H "Authorization: Bearer $KOVAC_API_KEY"

status moves staging → running → done (or failed/cancelled). counts breaks the total down by item status as they land; once staging finishes extracting, total_items gives you the real denominator for a progress bar instead of a number that keeps climbing.

A file’s folder path inside the zip becomes its path metadata (see The derived path below), and any sidecar files inside the zip are read automatically — this is the only one of the three ways in that resolves sidecars, because it is the only one where the caller does not already control each file’s metadata directly.

2. Folder upload

For a folder too large to zip conveniently, or a client that wants to show per-file progress as it goes: create an empty import, push files into it one at a time — each one the same request a single-file upload makes, just carrying an extra field — then close it.

# 1. Open the import
export IMPORT_ID=$(curl -s -X POST "$KOVAC_BASE/api/manage/collections/$COLLECTION_ID/imports" \
  -H "Authorization: Bearer $KOVAC_API_KEY" | jq -r '.import_id')

# 2. Push each file (repeat for every file in the folder)
curl -s -X POST "$KOVAC_BASE/api/manage/collections/$COLLECTION_ID/documents" \
  -H "Authorization: Bearer $KOVAC_API_KEY" \
  -F "file=@policies/handbook.pdf" \
  -F "import_id=$IMPORT_ID" \
  -F "path=policies" \
  -F 'metadata={"department": "hr"}'

# 3. Close it once every file has been pushed
curl -s -X POST "$KOVAC_BASE/api/manage/imports/$IMPORT_ID/close" \
  -H "Authorization: Bearer $KOVAC_API_KEY"

path is the file’s folder, relative to the root you are uploading — optional, and it becomes the same derived path metadata the zip method produces from the archive’s own folder structure. metadata is a JSON object, exactly like a single-file upload’s metadata field, validated against the collection’s schema on that same request — a typo in a field name is a 422 on that one file’s push, not a failure discovered later in a report.

Pushing into an import that has already been closed, cancelled or has finished answers 409 — the whole point of closing is that nothing can be added to it afterwards while the worker is finishing it up.

3. Batch manifest

For a script pulling documents from somewhere else — a legacy CMS, a ticket system, another vector store’s export — a manifest names one entry per document, each either a URL the platform fetches itself or a file attached to the same request:

curl -s -X POST "$KOVAC_BASE/api/manage/collections/$COLLECTION_ID/documents:batch" \
  -H "Authorization: Bearer $KOVAC_API_KEY" \
  -F 'manifest={
        "items": [
          {"source_uri": "kb-142", "url": "https://kb.example.com/articles/142.html",
           "path": "articles", "metadata": {"department": "support"}},
          {"source_uri": "kb-143", "part": "file0",
           "filename": "onboarding.pdf", "metadata": {"department": "hr"}}
        ]
      }' \
  -F "file0=@onboarding.pdf"

Each entry gives exactly one of url (fetched later, at claim time, through the same SSRF-hardened client web sources use) or part (the name of a multipart field attached to this same request, for bytes you already have locally) — giving both, or neither, is a 422 naming the offending entry. source_uri is yours to choose; it is what a citation from this document will name, so make it something a reader could actually trace back to. path and metadata work exactly as they do for a folder upload’s per-file fields, resolved and validated per entry before the manifest is even accepted — the whole manifest is rejected as a unit if any entry is structurally invalid, so you never end up with half a batch queued.

The response is 202 with {"import_id": "...", "kind": "batch"} immediately — local-file entries are staged synchronously inside this request, but URL entries are only fetched once the worker claims them, so a manifest of hundreds of URLs does not hold this request open.

Sidecar metadata

Zip imports only. A sidecar is a file inside the archive that describes metadata for other files rather than being a document itself. Four JSON shapes are accepted, plus a CSV — because four different systems people migrate from each produce metadata in one of these shapes, and requiring you to hand-convert would turn every migration into a scripting project.

A per-file sidecar: <filename>.metadata.json

The simplest shape: a flat JSON object of metadata for exactly the file it sits next to.

handbook.pdf
handbook.pdf.metadata.json

{ "department": "hr", "effective_date": "2026-01-01" }

A folder sidecar: _metadata.json

One file describing a whole folder: defaults apply to every file in that folder, and files overrides specific ones by name.

policies/
  _metadata.json
  handbook.pdf
  travel.pdf

{
  "defaults": { "department": "hr" },
  "files": {
    "travel.pdf": { "department": "finance" }
  }
}

Here handbook.pdf gets department: "hr" from defaults, and travel.pdf gets department: "finance" from its files override — the override replaces only the keys it names, so a file with several fields in defaults keeps the ones an override does not mention.

A folder sidecar, CSV: _metadata.csv

The same folder-level shape as a spreadsheet, for metadata that already lives in one. Exactly one column name is recognised for the file it describes: filename.

reports/
  _metadata.csv
  q1.pdf
  q2.pdf

filename,department,region
q1.pdf,finance,emea
q2.pdf,finance,amer

A cell for a declared number, boolean or tags field is parsed as that type (tags also accepts a semicolon-separated list, since that is what a spreadsheet actually produces); every other cell is left as text for the collection’s own schema to validate.

Onyx’s shape: .onyx_metadata.json

A straight filename-keyed map, for a migration out of Onyx: no defaults, just one entry per file.

{
  "handbook.pdf": { "department": "hr" },
  "travel.pdf": { "department": "finance" }
}

Bedrock migration

Amazon Bedrock knowledge bases write a metadataAttributes object next to each source object, either as a whole sidecar or nested inside one .onyx_metadata.json-style entry:

{
  "metadataAttributes": {
    "department": { "value": { "type": "STRING", "stringValue": "legal" } },
    "confidential": { "value": { "type": "BOOLEAN", "booleanValue": true }, "includeForEmbedding": true }
  }
}

This shape is read and translated automatically — department becomes the plain string "legal", confidential becomes the plain boolean true — except includeForEmbedding, which is parsed so it is never a validation error, then discarded with a per-item warning naming the field. Whether a field’s value is folded into the text an embedding model sees is a property of the collection’s own metadata schema (FieldSpec.embed; see Metadata and filters), not of any one document — honouring a per-document includeForEmbedding would let two documents in the same collection embed different field sets, an inconsistency retrieval can’t show you. If you need a field embedded, declare it as such on the collection once, rather than per Bedrock sidecar.

Precedence

A file’s final metadata is built by layering every sidecar that applies to it, broadest first, most specific last — later layers win, key by key, not whole-object replacement. In order:

1. The root folder’s sidecar defaults.
2. The root folder’s sidecar files/Onyx entry for this file, if any.
3. Each subfolder’s sidecar defaults, outermost to innermost.
4. Each subfolder’s files/Onyx entry for this file, if any.
5. The file’s own <filename>.metadata.json, if one sits next to it.

A worked example, where all four layers below disagree about department:

docs/
  _metadata.json                      # {"defaults": {"department": "legal"}}
  policies/
    _metadata.json                    # {"defaults": {"department": "hr"},
    │                                  #  "files": {"handbook.pdf": {"department": "hr-eu"}}}
    handbook.pdf
    handbook.pdf.metadata.json        # {"department": "hr-updated"}

For policies/handbook.pdf, the layers apply in this order:

Layer	Source	department
1	docs/_metadata.json defaults	legal
2	(no root-level files entry for this file)	—
3	policies/_metadata.json defaults	hr
4	policies/_metadata.json files["handbook.pdf"]	hr-eu
5	handbook.pdf.metadata.json	hr-updated

handbook.pdf ends up with department: "hr-updated" — the most specific sidecar available wins, but only for the keys it actually sets; a key none of the later layers mention keeps whatever an earlier layer gave it.

A broken sidecar (invalid JSON, a CSV missing its filename column) fails only the files it would have governed, reported with reason sidecar — every other folder in the same archive still imports normally.

The derived path

Every file’s folder — its position inside the zip, the path field on a folder-upload push, or a manifest entry’s path — is written to its path metadata automatically, so you can filter or facet on “everything under policies/” without declaring it by hand. It is a plain string extra unless you declare path as a field on the collection’s schema.

A sidecar (or an explicit metadata field, for the folder and manifest methods) that already sets path wins — the derived value only fills in when nothing more specific said otherwise. This is deliberate: path is a fact about where an archive happened to put a file, and a sidecar is allowed to correct it, the same way any other field can be overridden by a more specific layer.

The report: statuses and reasons

Every file gets exactly one row, whether it succeeded, was skipped, or failed — an import that reports 750 documents queued out of 800 without naming what happened to the other 50 is not a usable report.

curl -s "$KOVAC_BASE/api/manage/imports/$IMPORT_ID/items" \
  -H "Authorization: Bearer $KOVAC_API_KEY"

or, for a spreadsheet-friendly export of the same rows:

curl -s "$KOVAC_BASE/api/manage/imports/$IMPORT_ID/items.csv" \
  -H "Authorization: Bearer $KOVAC_API_KEY"

Each row’s status is queued, running, done, skipped, failed or cancelled; a skipped or failed row also carries a reason:

Reason	Status	What it means	Retry fixes it?
duplicate	skipped	Content identical to a document already in this collection (or to another file earlier in the same import).	No — this isn’t a failure; nothing to fix.
unsupported	skipped	The file’s extension isn’t one the platform parses. See Ingest a document and query it for the accepted list.	No — change the file, then re-upload it.
metadata	failed	The file’s resolved metadata failed the collection’s schema (a required field missing, a value of the wrong type). error_field names the field.	No — fix the metadata source (the sidecar, or the metadata you sent), then re-upload.
too_large	failed	The file is over the per-file upload ceiling (KOVAC_MAX_UPLOAD_MB).	No — the file itself needs to shrink or the operator needs to raise the ceiling.
archive_bomb	failed	The file is itself a zip-based container format (.docx, .pptx, .xlsx, …) whose declared inflated size is implausible for its compressed size.	No — the file is refused on its own; re-upload won’t change that verdict.
format_mismatch	failed	The file’s actual bytes don’t match its extension (a .pdf that is really a ZIP, an HTML error page saved as .docx).	No — the source file itself is wrong.
path	failed	The file’s folder path is unsafe (e.g. contains ..).	No — fix the path and re-upload.
sidecar	failed	A sidecar governing this file couldn’t be parsed (bad JSON, a CSV missing its filename column).	No — fix the sidecar and re-upload.
stage_error	failed	Something went wrong turning the file into a document before it was ever queued (a database error, or — a defensive case — an internally inconsistent queue row).	Sometimes — see below.
ingest_error	failed	The file was queued and claimed, but parsing, chunking or embedding it failed (a corrupt PDF, an embedding-provider timeout), or fetching a manifest URL failed.	Yes.
worker_lost	failed	The worker crashed or was restarted while this item was in progress, and it had already used up its retry attempts.	Yes.

Retry (POST /imports/{id}/retry) re-queues only items that were actually claimed at least once — that is exactly ingest_error, worker_lost, and the rare claim-time stage_error. Everything else above was refused at staging, before any worker ever touched it: the same file, unchanged, would fail identically a second time, so retrying it would do nothing. Fix the underlying problem (the file, its metadata, or its sidecar) and upload it again instead.

curl -s -X POST "$KOVAC_BASE/api/manage/imports/$IMPORT_ID/retry" \
  -H "Authorization: Bearer $KOVAC_API_KEY"

The response names how many items were retryable and were re-queued; the import’s own status returns to running if any were.

Limits

• File count — KOVAC_MAX_IMPORT_FILES, default 5000. Applies to a zip’s entry count and a batch manifest’s item count.
• Size — KOVAC_MAX_IMPORT_MB, default 2048 (2 GB). Bounds a zip upload’s total size and the extraction it triggers.
• Per-file size — the same KOVAC_MAX_UPLOAD_MB a single-file upload is bound by (default 50 MB); a file over it is reported too_large rather than the whole import being refused.

All three are operator-configurable and enforced identically whichever of the three ways in you use — the API and the worker that processes the import both read the same two KOVAC_MAX_IMPORT_* values, so what the upload accepts and what the worker is willing to extract never disagree.

Next: Errors & limits.