Skip to content

Judge Agent

Overview

The Judge Agent is an LLM-powered evaluator that automatically determines whether your agent under test meets defined success criteria. Instead of writing complex assertion logic, you describe what success looks like in natural language, and the judge evaluates each conversation turn to decide whether to continue, succeed, or fail the test.

After each agent response, the judge:

  1. Reviews the entire conversation history
  2. Evaluates against your defined criteria
  3. Decides whether to continue, succeed, or fail

Use Case Example

Let's test a customer support agent handling billing inquiries:

python
import pytest
import scenario
 
@pytest.mark.asyncio
async def test_billing_inquiry_quality():
    result = await scenario.run(
        name="billing inquiry handling",
        description="""
            User received an unexpected charge on their credit card and is
            concerned but polite. They have their account information ready.
        """,
        agents=[
            CustomerSupportAgent(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(criteria=[
                "Agent asks for account information to investigate",
                "Agent explains the charge clearly",
                "Agent offers a solution or next steps",
                "Agent maintains a helpful and empathetic tone",
                "Agent should not make promises about refunds without verification"
            ])
        ],
        max_turns=8
    )
    
    assert result.success

Configuration Reference

ParameterTypeRequiredDefaultDescription
criteriaList[str]No[]Success criteria to evaluate. Include positive requirements and negative constraints.
modelstrNoGlobal configLLM model identifier (e.g., "openai/gpt-4o").
temperaturefloatNo0.0Sampling temperature (0.0-1.0). Use 0.0-0.2 for consistent evaluation.
max_tokensintNoModel defaultMaximum tokens for judge reasoning and explanations.
system_promptstrNoBuilt-inCustom system prompt to override default judge behavior.
api_basestrNoGlobal configBase URL for custom API endpoints.
api_keystrNoEnvironmentAPI key for the model provider.
**extra_paramsdictNo{}Additional LiteLLM parameters (headers, timeout, client).

Writing Effective Criteria

Good criteria are specific, measurable, relevant, and actionable:

python
# Good - specific and measurable
scenario.JudgeAgent(criteria=[
    "Agent asks for the user's order number",
    "Agent provides a tracking link",
    "Agent offers to help with anything else",
    "Agent should not promise delivery dates without checking the system"
])
 
# Avoid vague criteria
scenario.JudgeAgent(criteria=[
    "Agent is helpful",           # Too vague
    "Agent does everything right", # Not measurable
])

Guaranteeing a Minimum Number of Turns

The judge looks at the conversation after every turn and may end the run. On turn 1, most criteria are unmet simply because the agent hasn't had a chance to act yet — and an eager judge can read "not met yet" as "failed" and end the run immediately.

minTurns (Python: min_turns) sets a floor: the first minTurns turns of a run are guaranteed to happen, because the judge is not offered the option to end the test on those turns. With minTurns: 4, turns 1–4 always run, and the earliest the judge can end the run on its own is turn 5. It pairs with maxTurns the way a floor pairs with a ceiling.

python
result = await scenario.run(
    name="refund negotiation",
    description="User wants a refund for a subscription they forgot to cancel",
    agents=[
        my_agent,
        scenario.UserSimulatorAgent(),
        scenario.JudgeAgent(criteria=[
            "Agent acknowledges the request",
            "Agent checks the account history",
            "Agent offers a resolution",
        ]),
    ],
    min_turns=4,  # turns 1-4 always run; earliest verdict on turn 5
    max_turns=10,
)

Things to know:

  • It's opt-in. Unset, nothing changes — the judge behaves exactly as today.
  • Explicit judgments still work. A scenario.judge() step in a script gets a real verdict even below the floor, and the final turn always gets one. The floor only stops the judge from volunteering a verdict early.
  • It only governs the judge. Red-team scenarios can still end early when an attack is confirmed successful, and scenario.succeed() / scenario.fail() script steps still end the run — those are deliberate decisions, not the premature "criteria not met yet" misread that minTurns prevents.
  • minTurns must be a non-negative whole number no greater than maxTurns — invalid values are configuration errors raised when the scenario is built. Zero is valid and behaves like an unset floor. Setting the floor and ceiling equal is also valid: the only verdict is the final-turn one.

Next Steps