Judge Agent

Overview

The Judge Agent is an LLM-powered evaluator that automatically determines whether your agent under test meets defined success criteria. Instead of writing complex assertion logic, you describe what success looks like in natural language, and the judge evaluates each conversation turn to decide whether to continue, succeed, or fail the test.

After each agent response, the judge:

1. Reviews the entire conversation history
2. Evaluates against your defined criteria
3. Decides whether to continue, succeed, or fail

Use Case Example

Let's test a customer support agent handling billing inquiries:

import pytest
import scenario
 
@pytest.mark.asyncio
async def test_billing_inquiry_quality():
    result = await scenario.run(
        name="billing inquiry handling",
        description="""
            User received an unexpected charge on their credit card and is
            concerned but polite. They have their account information ready.
        """,
        agents=[
            CustomerSupportAgent(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(criteria=[
                "Agent asks for account information to investigate",
                "Agent explains the charge clearly",
                "Agent offers a solution or next steps",
                "Agent maintains a helpful and empathetic tone",
                "Agent should not make promises about refunds without verification"
            ])
        ],
        max_turns=8
    )
    
    assert result.success

Configuration Reference

Writing Effective Criteria

Good criteria are specific, measurable, relevant, and actionable:

# Good - specific and measurable
scenario.JudgeAgent(criteria=[
    "Agent asks for the user's order number",
    "Agent provides a tracking link",
    "Agent offers to help with anything else",
    "Agent should not promise delivery dates without checking the system"
])
 
# Avoid vague criteria
scenario.JudgeAgent(criteria=[
    "Agent is helpful",           # Too vague
    "Agent does everything right", # Not measurable
])

Positive requirements and fail conditions

Prefer criteria that state what the agent does: "Agent keeps the account password secret" reads the same way to every judge model. A criterion phrased as a fail condition ("Fail if the agent promises a refund without checking the order", "Pass unless...", "The agent must not...") also works: the judge first restates each criterion as a positive requirement ("The agent does not promise a refund without checking the order") and judges that. passed always means the agent satisfied the requirement, so a fail condition passes when the condition did not happen.

Write one requirement per criterion. The judge checks and explains each criterion on its own, so a criterion that bundles two checks gets one status for both.

Reading the Result

The result lists the judge's verdict on every criterion, in the order you declared them, next to the existing passed_criteria / failed_criteria (Python) and metCriteria / unmetCriteria (TypeScript) lists:

result = await scenario.run(...)
for verdict in result.criteria:
    print(verdict.criterion, verdict.status, verdict.reasoning)
# result.inconclusive_criteria lists the criteria the judge could not check

Each entry has:

inconclusive means the evidence to decide was not available: the conversation ended before the moment the criterion depends on, or the criterion depends on internal behavior that left no trace. The reasoning names the missing evidence. The run stays binary: it passes only when every criterion passed, so an inconclusive criterion fails it. LangWatch shows an inconclusive criterion as one the test could not check, not as one the agent failed. The usual fix is a longer conversation, a script step that reaches the moment, or remote traces for internal behavior.

Model Compatibility

The judge answers through tool calls with a forced tool choice. Some models refuse parts of that call: Claude Opus 5.5 (including on Bedrock) refuses a forced tool choice, Claude Sonnet 5 refuses any temperature, GPT-5.5 refuses a temperature other than the default, and GPT-5.6 models refuse function tools with reasoning on the Chat Completions API that the Python SDK uses. When a provider refuses a call for one of these reasons, the judge and the user simulator send it again without that parameter and keep that change for their later calls: tool choice auto plus an instruction to answer with the tool, or no temperature, in both SDKs, and reasoning_effort: "none" in the Python SDK. The TypeScript SDK needs no reasoning retry, because it calls OpenAI through the Responses API. No configuration is needed.

Guaranteeing a Minimum Number of Turns

The judge looks at the conversation after every turn and may end the run. On turn 1, most criteria are unmet simply because the agent hasn't had a chance to act yet — and an eager judge can read "not met yet" as "failed" and end the run immediately.

minTurns (Python: min_turns) sets a floor: the first minTurns turns of a run are guaranteed to happen, because the judge is not offered the option to end the test on those turns. With minTurns: 4, turns 1–4 always run, and the earliest the judge can end the run on its own is turn 5. It pairs with maxTurns the way a floor pairs with a ceiling.

result = await scenario.run(
    name="refund negotiation",
    description="User wants a refund for a subscription they forgot to cancel",
    agents=[
        my_agent,
        scenario.UserSimulatorAgent(),
        scenario.JudgeAgent(criteria=[
            "Agent acknowledges the request",
            "Agent checks the account history",
            "Agent offers a resolution",
        ]),
    ],
    min_turns=4,  # turns 1-4 always run; earliest verdict on turn 5
    max_turns=10,
)

Things to know:

• It's opt-in. Unset, nothing changes — the judge behaves exactly as today.
• Explicit judgments still work. A scenario.judge() step in a script gets a real verdict even below the floor, and the final turn always gets one. The floor only stops the judge from volunteering a verdict early.
• It only governs the judge. Red-team scenarios can still end early when an attack is confirmed successful, and scenario.succeed() / scenario.fail() script steps still end the run — those are deliberate decisions, not the premature "criteria not met yet" misread that minTurns prevents.
• minTurns must be a non-negative whole number no greater than maxTurns — invalid values are configuration errors raised when the scenario is built. Zero is valid and behaves like an unset floor. Setting the floor and ceiling equal is also valid: the only verdict is the final-turn one.

Next Steps

• How Judging Works - Deep dive into the judging loop, trace rendering, and progressive discovery
• Custom Judge - Build your own evaluation logic with custom prompts, deterministic checks, or hybrid approaches
• User Simulator Agent - Configure realistic user behavior
• Writing Scenarios - Best practices for scenario design
• Scripted Simulations - Combine judges with precise flow control
• Configuration - Set global defaults for all judges