Optionalcfg: JudgeAgentConfigConfiguration for the judge agent.
Configuration for the judge agent.
Optionalcriteria?: string[]The criteria that the judge will use to evaluate the conversation.
OptionalmaxDiscoverySteps?: numberMaximum number of tool-calling steps for progressive trace discovery. Only applies when the trace exceeds the token threshold.
OptionalmaxTokens?: numberOptionalmodel?: LanguageModelOptionalname?: stringThe name of the agent.
OptionalspanCollector?: JudgeSpanCollectorOptional span collector for telemetry. Defaults to global singleton.
OptionalsystemPrompt?: stringA custom system prompt to override the default behavior of the judge.
Optionaltemperature?: numberOptionaltokenThreshold?: numberToken threshold for switching to structure-only trace rendering. When the full trace digest exceeds this estimated token count, the judge receives a structure-only view with expand_trace and grep_trace tools for progressive discovery.
import { run, judgeAgent, AgentRole, user, agent, AgentAdapter } from '@langwatch/scenario';
const myAgent: AgentAdapter = {
role: AgentRole.AGENT,
async call(input) {
return `The user said: ${input.messages.at(-1)?.content}`;
}
};
async function main() {
const result = await run({
name: "Judge Agent Test",
description: "A simple test to see if the judge agent works.",
agents: [
myAgent,
judgeAgent({
criteria: ["The agent must respond to the user."],
}),
],
script: [
user("Hello!"),
agent(),
],
});
}
main();
Factory function for creating JudgeAgent instances.
JudgeAgent evaluates conversations against success criteria.
The JudgeAgent watches conversations in real-time and makes decisions about whether the agent under test is meeting the specified criteria. It can either allow the conversation to continue or end it with a success/failure verdict.
The judge uses function calling to make structured decisions and provides detailed reasoning for its verdicts. It evaluates each criterion independently and provides comprehensive feedback about what worked and what didn't.