@langwatch/scenario
    Preparing search index...

    Function run

    • High-level interface for running a scenario test.

      This is the main entry point for executing scenario tests. It creates a ScenarioExecution instance and runs it.

      Parameters

      • cfg: ScenarioConfig

        Configuration for the scenario test.

        Configuration for a scenario.

        • agents: AgentAdapter[]

          The agents participating in the scenario.

        • description: string

          A description of what the scenario tests.

        • OptionalfetchRemoteTraces?: boolean

          Whether the judge fetches the remote traces produced by the agent under test from the LangWatch trace API and merges them into its evaluation.

          Enable this when the agent runs behind an HTTP endpoint that returns final text only: the adapter forwards AgentInput.propagationHeaders to the remote agent, the agent's own spans land in the same trace, and the judge reads the real tool calls, writes, and retrievals instead of claims in the transcript.

          Can also be set project-wide in scenario.config.js; this per-run value wins.

          false
        • Optionalid?: string

          Optional unique identifier for the scenario. If not provided, a UUID will be generated.

        • Optionallangwatch?: LangwatchConfig

          LangWatch reporting configuration. Takes precedence over LANGWATCH_API_KEY and LANGWATCH_ENDPOINT environment variables.

          Use this when running multiple scenarios concurrently for different projects to avoid race conditions from mutating process.env.

          await run({
            name: 'My scenario',
            langwatch: {
              apiKey: project.apiKey,
              endpoint: 'https://app.langwatch.ai',
            },
            agents: [...],
          });
        • OptionalmaxTurns?: number

          The maximum number of turns to execute.

          If no value is provided, this defaults to DEFAULT_MAX_TURNS.

          DEFAULT_MAX_TURNS

        • Optionalmetadata?: Record<string, unknown>

          Optional metadata to attach to the scenario run. Accepts arbitrary key-value pairs (e.g. prompt IDs, environments, versions). The langwatch key is reserved for platform-internal use.

        • OptionalminTurns?: number

          The minimum number of turns that must run before the judge may volunteer a verdict. With minTurns: 4, turns 1–4 always run and the judge can first end the test on turn 5 — its finish_test tool is withheld on earlier turns (ADR-005).

          Forced judgments always win over the floor: an explicit scenario.judge() step and the final maxTurns turn still deliver a terminal verdict even below the floor. The floor governs the judge only — red-team early exit and explicit succeed()/fail() script steps are unaffected.

          Must be a non-negative integer and must not exceed maxTurns; invalid values throw at startup. Zero is valid and behaves like an unset floor. When unset, behavior is identical to previous releases.

        • name: string

          The name of the scenario.

        • OptionalonAudioChunk?: (chunk: AudioChunk) => void

          Optional callback invoked for every audio chunk that flows through a voice adapter (both user-side and agent-side).

          Mirrors Python scenario.run(on_audio_chunk=...). Best-effort — if the hook throws, the scenario continues uninterrupted.

        • OptionalonVoiceEvent?: (event: VoiceEvent) => void

          Optional callback invoked for every VoiceEvent appended to the timeline (user_start_speaking, agent_stop_speaking, etc.).

          Mirrors Python scenario.run(on_voice_event=...). Best-effort — if the hook throws, the scenario continues uninterrupted.

        • Optionalscript?: ScriptStep[]

          The script of steps to execute for the scenario.

        • OptionalsetId?: string

          Optional identifier to group this scenario into a set ("Simulation Set"). This is useful for organizing related scenarios in the UI and for reporting. If not provided, the scenario will not be grouped into a set.

        • OptionalthreadId?: string

          Optional thread ID to use for the conversation. If not provided, a new thread will be created.

        • OptionaltraceWaitExtensionMs?: number

          Budget in milliseconds for the judge's one extra wait. When the traces are still incomplete after the settle-wait, the verdict call offers the judge a wait_for_traces tool: calling it waits this budget once more, then the tool is withdrawn and the judge must decide. Only used when fetchRemoteTraces is enabled.

          Can also be set project-wide in scenario.config.js; this per-run value wins.

          the resolved traceWaitTimeoutMs
        • OptionaltraceWaitTimeoutMs?: number

          Total time budget in milliseconds the judge waits at verdict time for remote traces to arrive and stabilize. Only used when fetchRemoteTraces is enabled. Mid-conversation judge calls never wait; the budget applies once, when a verdict is required.

          Can also be set project-wide in scenario.config.js; this per-run value wins.

          30000
        • Optionalverbose?: boolean

          Whether to output verbose logging.

          If no value is provided, this defaults to DEFAULT_VERBOSE.

          DEFAULT_VERBOSE

        • Optionalvoice?: VoiceConfig

          Per-run voice configuration (ADR-002). This is the carrier that reaches every call() via AgentInput.scenarioConfig — the STT/TTS providers the judge's transcription pass and the user-simulator's TTS pass read live here, NOT in a module global. An optional RunOptions.voice override seeds this at the run() boundary (options?.voice ?? cfg.voice ?? default); the resolved provider is always read off cfg.voice. See voice/config.ts#resolveVoiceConfig.

      • Optionaloptions: RunOptions

      Returns Promise<ScenarioResult>

      A promise that resolves with the ScenarioResult containing the test outcome, conversation history, success/failure status, and detailed reasoning.

      import { run, AgentAdapter, AgentRole, user, agent } from '@langwatch/scenario';

      const myAgent: AgentAdapter = {
        role: AgentRole.AGENT,
        async call(input) {
          return `The user said: ${input.messages.at(-1)?.content}`;
        }
      };

      async function main() {
        const result = await run({
          name: "Customer Service Test",
          description: "A simple test to see if the agent responds.",
          agents: [myAgent],
          script: [
            user("Hello, world!"),
            agent(),
          ],
        });

        if (result.success) {
          console.log("Scenario passed!");
        } else {
          console.error(`Scenario failed: ${result.reasoning}`);
        }
      }

      main();

    Settings
    Member Visibility