How Judging Works

Inside the built-in judge

The built-in judge is an LLM-powered evaluator that watches the conversation after each turn. It receives the full transcript plus OpenTelemetry traces collected during the agent's execution, and uses function calling to make structured decisions — either continuing the simulation or ending it with a per-criterion verdict.

This page explains what happens under the hood: the judging loop, how traces are rendered, how progressive discovery works for large traces, and the decision-making contract.

The Judging Loop

After every agent turn, the scenario runner calls the judge. The judge sees the full conversation so far, any OTel traces captured during the agent's execution, and the scenario criteria. It then decides whether to continue or finish:

                        ┌──────────────────────────────────┐
                        │         Scenario Loop            │
                        │                                  │
                        │  user() ──► agent() ──► judge()  │
                        │    │           │           │     │
                        │    ▼           ▼           ▼     │
                        │  "Help me"  "Sure,..."   ????    │
                        └────────────────────────────┬─────┘
                                                     │
                         ┌───────────────────────────┘
                         │
                         ▼
               ┌─────────────────────┐
               │   Judge receives    │
               │                     │
               │  - Full transcript  │
               │  - OTel traces      │
               │  - Criteria list    │
               │  - Scenario desc    │
               └────────┬────────────┘
                        │
                        ▼
               ┌─────────────────────┐
               │   Decision call     │
               │                     │
               │  continue_test ───────► loop continues
               │       or            │
               │  make_verdict       │
               └────────┬────────────┘
                        │
                        ▼
               ┌─────────────────────┐
               │   Verdict call      │
               │                     │
               │  finish_test ─────────► ScenarioResult
               │   (verdict +        │     success/failure
               │    per-criterion    │     + reasoning
               │    true/false)      │
               └─────────────────────┘

The judge is two-phase. A mid-conversation call is a decision only: has the conversation collected enough information to evaluate the criteria? The decision is expressed through two argument-free tools, continue_test and make_verdict — no reasoning field and no per-criterion schema, so the judge cannot pre-commit to pass or fail before it sees the full evidence. The decision prompt leans towards continuing while the conversation is still short, since scenarios exist to exercise multi-turn behavior.

The verdict is a separate call with its own prompt and one terminal tool, finish_test. It is entered from a make_verdict decision, from the last turn, or from an explicit judge step in a scripted simulation — the last two skip the decision call entirely. Below the min_turns floor the judge returns continue without any LLM call at all.

The loop repeats until a verdict lands or the maximum number of turns is reached.

What the Judge Receives

The judge is called with an AgentInput containing:

input.messages -- the full conversation history in OpenAI message format. This includes all user, assistant, and tool messages from every turn so far.
input.judgment_request -- present when a verdict is expected. May contain .criteria for inline overrides (used by scripted simulations).
input.scenario_state -- scenario metadata including the description, current turn number, and maximum turns allowed.

The built-in judge formats these into two main sections for the LLM:

Transcript -- Messages formatted as role: content pairs, giving the judge a clear view of the conversation flow.
OTel traces -- A digest of all spans collected during the agent's execution, showing what happened internally (LLM calls, tool invocations, errors, timing).

The trace digest is built from locally collected spans. When remote trace fetching is enabled, spans the agent under test reported to LangWatch are fetched and merged into the same digest before it is built (see Judging on Remote Traces below).

How Traces Are Rendered

The trace digest is the judge's window into the agent's internal behavior. The rendering strategy depends on the size of the trace.

Full Inline Mode (default)

When the rendered trace digest is under approximately 8192 estimated tokens, the judge sees every span's full details inline -- name, duration, all attributes, events, and errors. This works well for simple agents with a handful of spans.

Here is an example of what a full trace digest looks like:

agent.run (2.00s)
├── [a0b1c2d3] llm.call (400ms)
│     model: gpt-4
│     gen_ai.prompt: "What is the weather in Paris?"
│     gen_ai.completion: "Let me check the weather for you."
├── [c2d3e4f5] tool.fetch_weather (300ms)
│     tool.name: fetch_weather
│     tool.input: {"city": "Paris"}
│     tool.output: {"temp": 22, "condition": "sunny"}
└── [d3e4f567] llm.completion (500ms)
      gen_ai.prompt: "Summarize the weather report"
      gen_ai.completion: "The weather in Paris is sunny with a temperature of 22°C."

Every span includes its full attributes and content, so the judge can inspect prompts, completions, tool inputs/outputs, and errors without any extra steps.

Structure-Only Mode + Progressive Discovery

When traces exceed approximately 8192 estimated tokens -- common with complex agents that make many LLM calls or tool invocations -- sending the full trace would consume too much of the judge's context window. Instead, the judge receives a structure-only view showing just span names, durations, and hierarchy, with 8-character span IDs in brackets:

agent.run (2.00s)
├── [a0b1c2d3] llm.call (400ms, 1500 tokens)
├── [c2d3e4f5] tool.fetch_weather (300ms)
├── [d3e4f567] llm.completion (500ms, 800 tokens)
└── [e4f56789] failed.operation (100ms) ⚠️ ERROR: Connection refused
 
Use expand_trace(span_id) to see span details or grep_trace(pattern) to search across spans.

The judge then gets two additional tools to drill into the trace on demand:

expand_trace -- Expand one or more spans by ID (or 8-character prefix) to see full attributes, events, and content. The judge calls this when it needs to inspect a specific span in detail, for example to read the prompt sent to an LLM or the output of a tool call.

grep_trace -- Search across all span attributes, events, and content for a pattern (case-insensitive). Returns matching spans with context, limited to 20 matches. This is useful when the judge needs to find a specific keyword, API endpoint, or error message across a large trace without expanding every span individually.

The judge can call these tools multiple times in a loop before answering — in the decision phase before choosing continue_test or make_verdict, and in the verdict phase before finish_test. This keeps the initial context compact while allowing deep inspection when needed.

Judging on Remote Traces

With fetch_remote_traces / fetchRemoteTraces enabled (see Remote Traces for the setup), a fetch stage runs before the trace digest is built. The judge collects every distinct trace id stamped on the conversation's messages, one per turn, fetches those traces from the LangWatch trace API, filters out scenario infrastructure spans, deduplicates against locally collected spans, and feeds the remainder into the same span collector the digest reads. Remote spans then behave exactly like local ones: full inline rendering, structure-only mode, expand_trace and grep_trace.

Settle-wait timing

Remote spans take time to be ingested, so the fetch runs exactly once, right before the verdict:

Conversation turns never fetch. A decision call reads only the locally collected spans; remote trace arrival plays no part in whether the conversation continues.
Every verdict settle-waits first. Each pending trace is polled every second until it is complete, meaning it holds at least one remote span from the agent under test and every fetched span's parent span has arrived, all traces in parallel, under one shared trace_wait_timeout / traceWaitTimeoutMs budget (default 30 seconds). A failed poll retries until the deadline, so a transient API error does not end the wait early. When the budget runs out first, the judge keeps every span that did arrive and the digest gains a synthetic error span marking the trace as possibly incomplete.
The judge can ask for one extra wait. When the settle-wait ends with incomplete traces, the verdict call offers a one-shot wait_for_traces tool. Calling it re-arms the failed traces, retracts their synthetic error spans, and settle-waits once more under the trace_wait_extension / traceWaitExtensionMs budget (default: the wait budget itself). The tool is then withdrawn, so the re-entered verdict call must decide on the evidence at hand.

Voluntary verdicts and inconclusive outcomes

A verdict entered voluntarily, through a make_verdict decision, that comes back inconclusive continues the conversation: the judge decided the information might be there, looked properly, and found it was not yet. A verdict that was required, on the last turn or from an explicit judge step, is terminal either way. One exception makes broken setups fail fast: when not one trace of the run ever settled, more turns cannot produce trace evidence, so a voluntary inconclusive verdict is terminal too.

The remote traces rules

When remote fetching is enabled, this rule is appended to the verdict system prompt's rules section (identical in both SDKs):

Criteria about the agent's internal behavior (tool calls, database writes, API calls, retrievals) must be verified against the <opentelemetry_traces> section, not against claims in the transcript. If a span named langwatch.span_collection.error is present, read its reason: when no agent spans arrived, mark criteria that depend on internal behavior as inconclusive, never passed. When the trace is incomplete, criteria proven by the spans that are present may pass, and criteria whose evidence is missing stay inconclusive. Criteria about the conversation itself are unaffected by missing traces: judge them from the transcript as normal. Never mark internal-behavior criteria as passed based on the transcript alone.

The langwatch.span_collection.error span it refers to is synthetic, added once per failed trace with the failure reason. The reason separates the two outcomes the rule handles: no agent spans arrived at all, so nothing about internal behavior may pass, or the trace arrived but is still incomplete at the deadline, so the spans that are present still count as evidence. The rule is what keeps the judge from passing a trace-dependent criterion it never saw evidence for. The last sentence of the rule matters for agents that have not adopted trace propagation at all: their conversation-level criteria still judge normally on every run, and only internal-behavior criteria need the traces.

The decision prompt gets its own line (identical in both SDKs):

The agent's execution traces are fetched and verified at the verdict, after the conversation ends; they are not part of this decision. Do not continue the conversation only to wait for trace evidence, and do not end it early to see traces sooner.

The Decision Tools

The judge uses function calling to communicate its decisions. The tools split across the two phases:

`continue_test` (decision phase)

Let the conversation continue. The judge does not have enough information yet -- for example, the user has only asked one question and the criteria require a multi-turn interaction. Takes no arguments. Calling this returns control to the scenario loop, which runs the next user turn.

`make_verdict` (decision phase)

The conversation has collected enough information to evaluate the criteria. Takes no arguments -- deliberately, so the decision carries no judgment: the pass/fail evaluation happens in the verdict call, after the full evidence (including remote traces) is in. Calling this ends the conversation and moves to the verdict.

`finish_test` (verdict phase)

End the simulation with a verdict. This tool takes:

verdict -- One of "success", "failure", or "inconclusive". Determines the overall test outcome.
criteria -- A per-criterion result mapping each criterion to "true", "false", or "inconclusive". This powers the detailed breakdown in test output and the visualization dashboard.
reasoning -- A free-text explanation of why the judge reached this verdict. Included in test output for debugging.

The judge is expected to evaluate every criterion independently. A single failed criterion results in an overall "failure" verdict from the built-in judge.

`wait_for_traces` (verdict phase, one-shot)

Offered alongside finish_test only when remote fetching is on and the settle-wait ended with incomplete traces. Takes no arguments. Calling it settle-waits one extra trace_wait_extension / traceWaitExtensionMs period for the missing spans, then re-enters the verdict with the tool withdrawn: the judge is told the trace evidence is now final and must deliver its verdict. The tool description instructs the judge to call it only when a criterion genuinely depends on the missing spans.

Configuration

You can tune how trace rendering and progressive discovery behave:

python

import scenario
 
judge = scenario.JudgeAgent(
    criteria=["Agent uses the correct API endpoint"],
    # Traces under 8192 estimated tokens are rendered inline (default)
    token_threshold=8192,
    # Max expand/grep tool calls before forcing a verdict (default: 10)
    max_discovery_steps=10,
)

Remote trace fetching is configured on the scenario, not on the judge: