Golden sets

A golden set answers one question: on this collection, does retrieval find the right chunk? Ground truth here is generated and checked automatically, so it costs almost nothing to keep current, and it is cheap to re-run every time chunking, embeddings, or retrieval settings change.

This is a different golden set from the one Read traces and evals describes for offline agent evaluation — or rather, the same accepted items feed both. Generation and review produce retrieval ground truth (a question plus the chunk ids that answer it) stored per collection; exporting it writes the same loadable YAML shape kovac eval-offline already reads, so one accepted set backs both the per-collection retrieval metrics on this page and the agent-level gate that runs before a deploy.

Exporting a set for the agent-quality gate is still a kovac CLI operation run against the deployment, not a call this docs surface’s API key can make — the same kind of operator step as kovac ingest. Generating a set has two routes instead: starting a run from the collection’s Golden set tab, a management API call like anything else under /api/manage, or running kovac golden-generate directly. Either route can add unanswerable questions (the negative slice). Reviewing candidates and reading the retrieval-quality history are management API endpoints too, callable with a tenant key the same way as anything else under /api/manage.

1. Generate a set

Both routes sample the collection’s own chunks, generate candidate questions, run them through the same four checks, and enrich whatever survives with a set of acceptable chunk ids — the difference is who starts the run and what it costs. The UI is the faster path for a one-off run with a dollar limit; reach for the CLI when you need to set the number of sampled chunks and the number of questions per chunk separately, or to script generation as part of a larger job.

From the Golden set tab

Knowledge → the collection → Golden set → Generate golden set opens a dialog. Its main inputs are below. The Answers and Questions sections, covered after the CLI, tune what is generated:

Input	Default	Notes
Candidates	200 (1–500)	one candidate question per sampled chunk, including any unanswerable ones
Generation model	the registry default, if priced	writes the questions
Judge model	a different model from the generator, where one exists	decides each candidate’s acceptable chunk ids
Maximum spend	$10	leave it empty for no limit

Both model pickers only offer priced chat models — a model with no token rates set cannot be chosen here, and starting a run against one directly through the API is refused outright (a 422 naming the field). “No rates” never means “free”: a genuinely free or self-hosted model gets an explicit rate of 0, which is valid and appears in the pickers normally. If the model you want isn’t listed, set its rate first in the Models screen.

Generation and judging are priced at their own models’ rates, and the dialog shows a cost estimate that recomputes as you change any input. Take that number as a starting figure, not a measurement: it’s built from representative prompt sizes rather than anything measured from a real call, which is exactly why every run — once finished — reports its estimate next to what it actually spent, so the constants behind the estimate can be corrected against real numbers over time rather than trusted forever. The judge side of that estimate usually matters more than the generation side: enrichment makes one judge call per retrieved chunk per accepted candidate, so the judge term scales with the collection’s rerank_top_n where the generation term does not — for any realistic rerank_top_n (5–10), the judge model’s price moves the total more than the generation model’s does. Choosing the same model for both roles is allowed and only warns, never blocks — a model grading questions it just wrote agrees with itself more than an independent judge would, but it’s your money and your call.

The maximum spend, when set, is enforced from real spend as the run executes — actual token usage from each model call, accounted in Decimal, added up as it goes — not from the pre-run estimate. The instant accumulated spend reaches the cap, the run stops immediately. A capped run is a normal, successful outcome, not a failure — but a candidate isn’t stored until it has been fully enriched, so the run keeps only whatever it had already stored by the time the cap tripped, which can be nothing if the cap fires early. The panel names what it kept, or says plainly when that’s nothing.

The tab shows a live panel for as long as a run is in flight — the stage it’s in and how many candidates it has produced so far — and stays on screen after you navigate away and back, or reload the page: it rediscovers the collection’s most recent run rather than depending on having just started it. A finished run shows its actual spend next to what it was estimated to cost, side by side, which is what lets the estimate’s underlying per-token constants be corrected against real numbers over time.

Occasionally a run’s executor crashes or otherwise stops reporting progress before it reaches a final outcome; the panel then shows the run as stale rather than leaving it “running” forever. A stale run is neither a success nor a confirmed failure — candidates are written in one batch near the very end of a run, so one that goes stale during its long generation/judging stretch (the common case) typically has nothing saved yet, and the panel says so rather than promising either outcome. The review queue below is the authoritative answer for what, if anything, actually landed.

Whichever way it finishes, generation never auto-accepts anything — candidates land proposed, same as the CLI path, and still need a human decision at Review (below) before they enter the measured set.

The dialog can add the negative slice too: set Unanswerable questions in its Questions section (see Question settings below). From the CLI, kovac golden-generate --negatives N does the same, through the same code.

From the CLI

kovac golden-generate handbook --sample 100 --per-chunk 2 --negatives 20

This samples up to --sample chunks from the collection (spread evenly across documents and sections, not first-come), asks the generation model for --per-chunk candidate questions per sampled chunk, runs every candidate through four checks (below), and enriches whatever survives with a set of acceptable chunk ids before storing it — accepted, rejected, and still-pending-review alike — for a human to look at. A separate --negatives count of unanswerable questions is generated alongside it from the collection’s section headings.

This command has no model flags of its own. The generation model resolves through the platform’s one utility-model default (the registry’s is_utility_default row, or failing that its chat default) — there is no environment variable and no hardcoded fallback behind it. The judge model resolves through almost the same chain, with one extra preference ahead of it: the registry’s own judge default (is_judge_default, a chat-only flag an operator sets on the Models screen, independent of is_utility_default), falling through to that same utility chain only when no row is flagged. With no judge default set — the common case on a freshly configured deployment — both models resolve identically, so running this command generates and judges with the same model: it prints what it resolved and warns about that every time, since the two cannot be anything else here —

generator model: 'openai:gpt-5-mini'   judge model: 'openai:gpt-5-mini'
warning: the generator and judge both resolve to 'openai:gpt-5-mini' -- with no CLI flag of their own, both fall back to the platform's one utility default. A model grading questions it wrote agrees with itself more than it should; flag a different model is_utility_default (or pass an explicit judge where the API supports one) for a real run.
generated:  184
  rejected (deictic): 12
  rejected (duplicate): 9
  rejected (leakage): 31
  rejected (not-standalone): 4
accepted for review: 128
negative slice: 18
document coverage: 94% of the collection's documents

For a real run, use the Golden set tab instead, or flag a judge default on the Models screen. The generation model and judge model pickers on the tab (above) each take an explicit choice; flagging a chat row is_judge_default on the Models screen gets the same independent-judge effect for THIS command too, with no per-run picker at all — once that flag is set, this command’s own judge model resolution stops matching the generator’s by construction, and the warning above stops printing.

If the registry has no model flagged is_utility_default and no chat default either, the command fails outright before generating anything — set one on the Models screen first. A resolvable model with no working credential fails differently, and is easier to miss: zero candidates generated from a non-empty sample of chunks is treated as a configuration error (the command exits non-zero and names the model it tried), so check that model’s stored key in the Models screen if a run reports zero candidates.

Answer settings

Every accepted question carries a reference answer, and the answer is written last: after the question has been generated, checked, and matched to every chunk that acceptably answers it. The model that writes it reads the question together with all of those chunks (or only the one the question came from, if you ask for that), so the answer can be complete rather than a few sentences drawn from a single passage. It states the direct answer first, then the reasoning and the conditions the passages give, and it says what the passages do not cover instead of filling the gap. It is written in the language of the passages.

The Answers section of the generate dialog controls how:

Setting	Choices	Default
Length	Short, Medium, Long, or Mixed with a percentage for each (they add up to 100)	Mixed: 20% short, 50% medium, 30% long
Style	Plain (prose only), Structured (paragraphs, plus lists or steps where the passages have them), Steps (numbered steps for procedures)	Structured
Explanation	Facts only, Facts and why, or Facts, why and caveats	Facts, why and caveats
Write from	All acceptable passages, or only the source passage	All acceptable passages
Mark citations	Add a bracketed passage number after each claim	Off
Answer model	Any priced chat model	Same as the generation model

Short means one or two sentences (up to about 40 words), medium one paragraph of roughly 60 to 120 words, and long about 150 to 300 words. The length is a ceiling the model is asked to stay within, not a quota: if the passages support less, the answer is shorter. We never cut an answer off afterwards, and one the model could not finish is not kept: the item keeps its draft answer instead. With a mixed length, items are given their length in a fixed rotation, so a run lands on the percentages you set to within one item.

The cost estimate includes one answer call per candidate, priced at the answer model’s rates. If an answer cannot be written for an item, the item keeps the draft answer written with its question, and the finished run reports how many fell back. Each item remembers the settings its answer was written with, and the review panel shows them in plain words, for example “Long · Structured · from 4 passages”.

Question settings

The Questions section of the generate dialog sits beside Answers and controls what the questions are like. Every setting has a default, so a run you start without touching them still works. As with answers, the model is asked to follow each target but may decline one a passage cannot support (a “how do I” question from a paragraph that only states a fact); the item then keeps the type the model actually wrote, and a run’s mixes are followed on the targets, so they hold even when some questions fall back.

Setting	Choices	Default
Question types	Let the model choose, or a mix over factual, how-to, why, comparison, troubleshooting, definition, list and numeric (they add up to 100)	Let the model choose
Difficulty	A mix of easy wording (shares obvious keywords), paraphrase and multi-hop (needs two facts)	30% / 50% / 20%
Phrasing	A mix of full questions, search keywords (two to six terms, no question mark) and casual (informal, with a typo or a missing accent)	60% / 25% / 15%
Persona	A line describing who is asking, such as “a nurse on a night shift” (up to 300 characters)	none
Language	The same as the documents, all questions in one language, or a share of them in one language	The same as the documents
Unanswerable questions	The percentage of the candidates that the collection should not be able to answer (up to 30)	0%
Questions that need two documents	The percentage of the answerable questions that need passages from two different documents (up to 30)	0%

Each mix is spread over the run in a fixed rotation, so it lands on your percentages to within one question. A question asked in a different language from its documents is answered in the question’s language.

Unanswerable questions are taken out of the Candidates count, not added to it: 100 candidates with 10% unanswerable is 90 answerable questions and 10 unanswerable ones. They are written from the collection’s section headings only, then each is searched for and judged against the best retrieved passages, so one the collection does answer is stored as rejected (reason “answerable”) rather than kept as a false negative. They have no answer and no acceptable chunks. The estimate counts the judge calls this takes, which scale with the collection’s retrieval depth.

Questions that need two documents pair a sampled passage with its most similar passage from a different document, then ask for a question that needs both. Both passages count as correct from the start, the question is labelled multi-hop, and its answer is written from both, whatever the answer settings say about writing from one passage. When no passage of another document is similar enough, that passage keeps its ordinary question, and the finished run reports how many fell back.

Keyword phrasing is allowed to share more of its words with the passage than a full question is: a search-box query is made of the document’s own nouns, so the leakage check is looser for it. The standalone check reads it as a query, not a question, and the other checks are unchanged.

Each item remembers its settings, and the review panel shows them, for example “How-to · keywords · mk · Long · Structured · from 3 passages”. The finished run reports how many unanswerable questions were stored and how many were rejected.

Scope

By default a run asks questions from every active document in the collection. The Scope section of the generate dialog narrows that, and changes how strictly near-duplicate questions are dropped. Every setting is optional, and the ones you set all apply together.

Setting	Choices	Default
Documents	Pick documents from the collection	all
Folders	Pick folders; a folder includes the folders under it	all
Only documents added since	A date; documents added on or after it (not edited since, since a metadata or status change would otherwise count)	any date
Metadata filter	Advanced: a filter on document metadata, in the same form search filters use, as JSON	none
At most N items per document	1 to 100	no limit
Skip questions similar to existing items	On or off	On
Similarity threshold	How alike two questions’ words must be before the newer one is dropped, from 0.3 to 0.95	0.6

The dialog shows how many passages the scope leaves to ask from, and refuses to start a run when there are none: widen the scope, or check that its documents are active. A metadata filter that does not fit the collection’s fields is refused with a message naming the field. A document you picked that has since been deleted is ignored.

A run reads the collection as the person who starts it. A document whose access settings hide it from you is never asked about, never paired into a two-document question and never becomes a correct answer, even when you name it in the scope, and the passage count leaves it out. Passages too short to ask about (a stray heading, a page number) are not counted either.

At most N items per document stops a long document from taking most of the set: once a document has N questions in the run, the rest are asked from the others. The run may then produce fewer questions than you asked for.

Skip questions similar to existing items compares each new question with every question already in the collection’s golden set, whatever its status, as well as with the others in the run. A question that matches is rejected as a duplicate, and the finished run reports how many of the duplicates matched an existing item. Turn it off to generate a second, independent set over the same documents.

Passages that a collection’s boilerplate rules exclude never become chunks, so they are never asked about and there is nothing to set.

Presets

A preset is a named, saved set of everything in the generate dialog: answers, questions and scope. Choose one under Start from at the top of the dialog and every section fills in; change anything you like before you start. Save as preset stores the dialog’s current settings under a name (up to 80 characters), and asks before overwriting a name that exists. Delete preset removes one.

Presets belong to a collection, up to 50 per collection, and are visible to everyone who can read it. Saving or deleting one needs the editor role and write access to the collection. A preset stores the settings, not the number of candidates, the models or the spend limit, which you choose for each run. Forge can start a run from a preset by name, and can read the list, but cannot save or delete one.

A preset whose saved scope can no longer be read in full (after an update changes what a scope can hold) is marked needs saving again under Start from. It fills in what still reads, but no run starts from it until you check its scope and save it again: a scope that lost a part would otherwise ask about more of the collection than the preset names.

Rewrite answers

Already have a golden set whose answers are short or incomplete? Rewrite answers, on the Golden set tab, writes them again with the settings you choose, without touching the questions or the chunks that count as correct.

The dialog asks which items to include (proposed, accepted, or both), the answer settings above, and a maximum spend, and shows the number of items and a cost estimate before anything runs. Items you edited by hand are left alone unless you tick Also rewrite answers I edited. Questions that the collection should not be able to answer have no answer to rewrite and are skipped.

A rewrite runs like a generation: a live panel shows its progress, the spend limit is enforced from real spend, and a run that reaches the limit keeps every answer it had already written. Because each answer is stored as soon as it exists, nothing is lost if a run stops partway. The finished run reports how many answers were rewritten and how many items were skipped or kept their old answer, and why.

A rewritten item goes back to proposed, because a person has not read the new text yet; accept it again at Review. Only one generation or rewrite runs on a collection at a time.

To rewrite a single answer, open the item in the review queue and choose Regenerate answer. It uses the settings last used for the collection, or the defaults, replaces the answer straight away, and asks first when the item was edited by hand.

What each check rejects, and why

Every rejected candidate is kept, labelled with the check that rejected it — never silently dropped. That is what makes the split above audible rather than just a number.

• deictic — the question only makes sense with the source passage in view (“what does the above table show?”, “is this section correct?”). A real user asking a search box has no passage in front of them.
• leakage — the question reuses enough of the source chunk’s own rare vocabulary that lexical search would match it trivially, regardless of whether the question is otherwise a fair paraphrase.
• duplicate — near-identical to a question already accepted from this run or already in the collection’s golden set (by word overlap, not meaning; the threshold is a scope setting), which would inflate the item count without adding real coverage.
• not-standalone — an LLM judge’s second opinion on standalone phrasing, run only on what the free deictic check already let through.
• answerable — negative-slice only: a question generated as “unanswerable” that a real retrieval pass shows the collection actually answers. It is rejected rather than kept as a false negative.

A very low rejection count is a red flag, not a compliment. It means the checks are not catching much, not that the generation model produced an unusually clean batch. If a run comes back with almost nothing rejected, look at the checks before trusting the accepted set.

2. Review

Generation writes every candidate as proposed; only a review decision moves one to accepted, and only accepted items are exported or scored. So nothing reaches the measured set without a decision — but the platform cannot tell you whose decision it was unless you look, and that distinction matters more than it sounds.

Knowledge → the collection → Golden set is the review screen. It lists every candidate with its ground-truth chunk count, difficulty tier, generation provenance (gen_model, judge_model, prompt_version) and current status, alongside the rejection split so you can see the automatic checks actually fired. Accept, edit or reject each item; accepting records your identity as its reviewer.

The screen also warns you when accepted items carry a reviewer ending in :machine-pre-review. A freshly generated set is often pre-filtered by a model before a person sees it — useful, since it turns ~200 candidates into ~80 worth reading — but a model’s accept is a suggestion, not a review. Those items are accepted, exported and scored like any other, so the banner is the only thing standing between “a machine liked this” and “a person signed this off”. Confirm or overturn them; the count drops as you go.

Everything on that screen is also available over the management API:

curl -s "$KOVAC_BASE/api/manage/collections/$COLLECTION_ID/golden-set?status=proposed" \
  -H "Authorization: Bearer $KOVAC_API_KEY" | jq '.items[] | {id, question, chunk_ids, difficulty}'

rejected on the same response is the rejection split above, grouped by reason — the same audit trail the CLI printed, readable at any later point without re-running generation.

Accept, edit, or reject one item:

curl -s -X PATCH "$KOVAC_BASE/api/manage/collections/$COLLECTION_ID/golden-set/$ITEM_ID" \
  -H "Authorization: Bearer $KOVAC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"status": "accepted"}'

Reviewing requires the editor role or above — it decides what every later quality claim gets measured against, and the generation that fed it cost real model calls. Listing the queue and reading the rejection split needs only a valid key, any role, so a viewer sees the set and its numbers without being able to move an item.

Supplying a question alongside status edits the wording and marks the item human-edited (provenance for later, per F6.8) rather than accepted as generated. chunk_ids — the acceptable-chunk set — is not editable through this endpoint yet; correcting ground truth today means rejecting the item and adding a corrected one.

Why ground truth is a set, not a single chunk id

Every accepted item’s ground truth is a set of chunk ids, not one. Marking exactly one chunk correct means that whenever another chunk in the collection also answers the question — routine in any real corpus, and more common once chunking gets structure-aware — a retrieval run that returns that other, equally correct chunk is recorded as a miss. That penalises retrieval for behaving correctly, and it biases every measured number against exactly the recall-improvement work (parent-child retrieval, contextual retrieval, adjacent-chunk expansion) those numbers are meant to guide.

3. Export for the agent-quality gate

kovac golden-export handbook --name dogfood

Writes every accepted item to evals/golden_set.dogfood.yaml, in the same shape kovac eval-offline has always read — so kovac eval-offline and the F9.8 eval-run API pick up the accepted set unchanged, no format translation, no separate maintenance. proposed and rejected items are never exported: the first has not been reviewed, the second was explicitly turned down, and either would put an unvetted or known-bad question into a gate that decides whether an agent version is safe to ship.

If you promote a failing production trace to the golden-set dataset from inside Langfuse (the one-click curation loop), pull it back into the review queue with:

kovac golden-import-promoted handbook

A promoted item arrives as proposed, carrying its originating trace id, but with no retrieval ground truth — a Langfuse dataset item has no chunk-id set to give it. A human supplies the acceptable chunk ids the same way as any other item, at review.

4. Measure

kovac eval-retrieval-metrics handbook --top-k 5

Scores every accepted, answerable item across four retrieval arms:

Arm	What it isolates
lexical	BM25 only — also the leakage self-check, below
dense	vector similarity only
hybrid	RRF fusion of both, at the collection’s configured weight
hybrid_rerank	hybrid, then cross-encoder reranked

For each arm: hit rate@k (did any acceptable chunk appear in the top k), MRR (reciprocal rank of the first acceptable chunk), NDCG@k, precision@k, and recall@k — all computed against the accepted item’s full chunk-id set, not a single answer.

The negative slice is scored separately — declined rate, the share of unanswerable questions whose best retrieved candidate fell below the collection’s own grade threshold — and is never averaged into the answerable arms’ hit rate. There is no correct chunk for an unanswerable question to hit; folding it in would drag every arm’s score down by however large the negative slice happens to be, while measuring nothing real.

By default the run is stored (--no-store to skip), so the same collection’s history is queryable later without re-running anything:

curl -s "$KOVAC_BASE/api/manage/collections/$COLLECTION_ID/retrieval-runs" \
  -H "Authorization: Bearer $KOVAC_API_KEY" | jq '.runs[] | {run_name, created_at, metrics}'

Every stored run carries its own leakage verdict alongside the four arms’ numbers (below), so a later trend read sees the same self-check a human running the CLI saw at the time.

Evaluations → Retrieval quality renders that history per collection: the newest run’s four arms with a signed delta against the previous run, the negative slice on its own line, the leakage verdict in whichever of its three states applies, and the same arms broken out per difficulty tier. The per-tier breakdown is not decoration — read the paraphrase tier’s leakage check in preference to lexical-easy’s or the overall figure, because lexical-easy questions share keywords with their source chunk by definition, so a strong lexical arm there is the label working as intended rather than evidence of a problem.

Reading results by setting

Under the difficulty table, Break down by on the retrieval-quality page groups the same run’s items by how they were generated and shows the four arms for each group: question type, phrasing, question language, answer length, and whether the question needed one document or two. Pick one to see, for example, whether keyword-style questions retrieve worse than full questions, or whether two-document questions are where the misses are.

The breakdown is read from the settings each item remembers, so it costs no extra searches, and only the dimensions your items actually vary in are offered. A group with fewer than three items is left out, since a rate over two questions says nothing, and the page says how many items it left out. Items generated before settings were recorded count as “unspecified”.

The leakage self-check

The lexical arm exists to catch a specific failure in the set itself, not to prove lexical retrieval is good. A model shown a chunk and asked to write a question about it will, left unchecked, write that question in the chunk’s own distinctive words — and BM25 then matches it almost by definition, regardless of collection or model. If lexical’s hit rate scores close to hybrid‘s, the questions are leaking their source chunk’s vocabulary, and every arm’s numbers on that set are inflated — not just lexical’s. The command reports this as an explicit verdict next to the four arms’ numbers rather than a hidden pass/fail, precisely so a good-looking hybrid score is never read at face value without checking whether the set earned it.

That inference has a precondition, and the verdict states it. hybrid is RRF fusion of dense and lexical, so when the dense arm is weak, hybrid collapses towards lexical by construction — whatever the questions look like. Terminology-dense corpora do exactly this: nobody paraphrases “Article 6”, so dense retrieval has little to add and a narrow lexical-to-hybrid gap says nothing at all about leakage. The verdict therefore has three states, not two:

Verdict	What it means
leaking	The gap is narrow and dense is competitive — the questions really are reusing their source vocabulary
inconclusive	Dense trails lexical by more than the competitiveness margin, so the gap cannot answer the question either way
healthy	The gap is wide enough, with a competitive dense arm

An inconclusive verdict is not a soft “probably fine”. It means this particular check has no signal on this collection, and reporting “leaking” there would push you to tighten generation against something that cannot respond to it.

What still works when the arm gap is inconclusive is the direct measure, reported on the same line: for every item, the share of the question’s IDF mass that it shares with the chunk it was generated from, summarised as a p25/median/p75/p90/max distribution. It reads the golden-set text itself and never touches retrieval, so no amount of weak dense recall can confound it. On a leaking set the median sits high; a healthy set’s p25 is at or near zero.

What this set measures, and what it does not

It measures retrieval. Ground truth is free, the metrics are deterministic, and they are cheap to re-run on every chunking change — which is the point.

It does not measure whether real users are served. Generated questions spread evenly over chunks; real questions cluster on a few. Treat it as the best available proxy until a knowledge-gap report built from real traffic is available.

Next: Connect a source.