Read traces and evals

Every run leaves a trace: the model calls, the tool calls, the passages retrieved before and after re-scoring, and any guardrail decision. Scores sit on top of that — automated judgements of whether an answer was grounded, relevant, and reached by sensible tool use.

Together they answer the two questions you actually have about an agent in production. Why did this one answer come out like that? is a trace question. Is this agent getting better or worse? is a scores question. Reaching for the wrong one wastes time.

Traces and scores are read in the platform, not over the published API.

Where to look

Open an agent on the Agents screen. Four of its tabs matter here.

Traces lists this agent’s recent runs and drills into one. A trace expands into its observation tree — every model call, tool call and retrieval nested under the step that caused it.

Monitor answers “is it healthy” at a glance: runs in the last 24 hours, error rate, 95th-percentile latency, and model cost over the same window, above a table of individual runs.

Evaluation shows this agent’s automated scores over the last 30 days as a trend, plus the offline evaluation runs it has been through.

Details carries the version history. When a trend changes shape on a particular day, this is where you find out what was deployed.

The tenant-wide Monitoring and Evaluations sections are the same data across every agent, which is where to start when you do not yet know which agent is misbehaving.

Reading a trace

Start at the top and go down. The tree is ordered, and the answer is almost always visible before you reach the model call that produced the text.

For a retrieval question, the useful detail is that each retrieval is recorded twice — the candidate set as first gathered, and the set that survived re-scoring. Comparing them separates two failures that look identical from outside:

• The right passage is absent from both sets. Retrieval missed it. The document may not be indexed, may be in a collection this agent is not attached to, or may be worded too far from the question.
• The right passage is in the first set but not the second. It was retrieved and then ranked away. That is a re-scoring outcome, and rerank_top_n is the lever — see Collections & knowledge.

Guardrail triggers are flagged inline in the tree, with the category that fired. A blocked run is not an HTTP error: the run ends with a refusal message in the stream, and the trace is where you see which category caused it.

Online scores

Production runs are judged automatically as they happen, on three dimensions:

• Hallucination — whether the answer is grounded in what was retrieved, rather than invented around it.
• Relevance — whether it answered the question that was asked.
• Tool-use correctness — whether the tools it reached for were the right ones for the task.

These are model judgements, not ground truth. Treat a single low score as a pointer to a trace worth reading, and the trend as the actual signal. A score that drops on the day you changed a system prompt is telling you something; one bad score on a Tuesday is not.

Online evaluation sampling

Judging every answer costs up to three judge calls per answer, and more when the answer used several tools, because the tool-use judge grades each tool-calling step. You can choose what share of answers the judges grade instead: at 30%, about three answers in ten are graded and the rest get no online scores. The rate is a whole percent from 0 to 100, and 0 turns online judging off.

A rate can be set in three places, and the most specific one wins:

1. The agent or workflow. The agent editor and the workflow’s settings have an Online evaluation field. Leave it on the workspace rate, or give the agent or workflow a rate of its own.
2. The workspace. A workspace admin sets it on the Evaluation tab of the workspace’s page under Tenants, which also estimates how many judge calls a day that rate means, from the last 7 days of runs.
3. The platform default, set by an operator on the Evaluations page. Until one is set, the default is 100%: every answer is graded.

The agents inside a workflow use the workflow’s rate, not their own, because a workflow’s run is one conversation.

The choice is made per run, from the run’s own id, so a run is either graded or not: every judge sees the same runs, and a lower rate grades a subset of the runs a higher one would. A changed rate needs no redeploy and takes effect within a minute.

Beside the online scores, a line says how many answers they rest on, for example “Based on 12 of 40 answers that used your documents”. It counts only answers that searched or cited your collections, so an agent that never uses them shows no line. An answer that was not graded is never counted as a failure, in the scores or in the feedback that ranks passages for review.

Offline evaluation

Offline evaluation runs an agent against a curated golden set — questions with known good answers — and judges the whole run at once, on grounding and on whether the right context was retrieved and used. It produces a gate verdict: pass or fail.

The difference from online scoring is what it is for. Online scores tell you how the agent is doing on real traffic, after the fact. An offline run tells you whether a change is safe to ship, before you ship it — same questions, same expectations, run against the new version.

Results appear on the agent’s Evaluation tab and in the tenant-wide Evaluations section, each showing the verdict and the judge scores behind it.

Answer completeness

An offline run also scores answer completeness: how much of the reference answer in the golden set the agent’s answer actually covers. The judge splits the reference into its key points, including any explanation and caveats, and scores the share the answer states correctly, lower for anything it contradicts. Extra correct detail and different wording do not cost anything. For a question the documents cannot answer, the reference is empty, and the answer scores 1 if the agent says it does not know and 0 if it answers anyway.

This score is shown with the run but does not count towards the pass or fail verdict yet, and a run that finishes without it still gets its verdict. It works best with full reference answers, so a set whose answers are one-line fragments will score low however good the agent is. Choose how full the answers are in the golden-set answer settings.

Workflows

A workflow has the same Traces, Monitor and Evaluation tabs as an agent. One run is one trace, with a span for each agent that took a turn, so you can see which agent made which call. The online scores judge a workflow’s answers as they judge an agent’s.

The Monitor tab adds Cost by agent (24h): each agent’s model calls, tokens and cost inside the workflow. An offline evaluation can run against a workflow, and its result is listed on the workflow’s Evaluation tab. An evaluation answers each question in a single run, so an item whose path reaches an approval (requires_approval on an edge or tool) pauses there and is not scored, and the evaluation fails naming it; keep approvals off a workflow you evaluate. See Workflows.

When the data is missing

Observability is deliberately not on the critical path. If the trace backend is unavailable, the Traces, Monitor and Evaluation views degrade to an empty state with a banner explaining why — and nothing else stops. Agents keep running, collections keep ingesting, and every management action still works.

So an empty Traces tab has two possible meanings: no runs yet, or the banner. Read the banner before concluding the agent is idle.

Next: Golden sets.