Skip to content

Getting Started: Your First Scenario

Installation

Install the Scenario SDK and a test runner (pytest or vitest):

python
uv add langwatch-scenario pytest

Quick Start

Create your first Scenario test. This example demonstrates testing a vegetarian recipe agent:

1. Create Your First Test

python
# my_vegetarian_recipe_agent.py
import pytest
import scenario
import litellm
 
# Configure the default model for simulations
scenario.configure(default_model="openai/gpt-4.1")
 
@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_vegetarian_recipe_agent():
    # 1. Create your agent adapter
    class RecipeAgent(scenario.AgentAdapter):
        async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
            return vegetarian_recipe_agent(input.messages)
 
    # 2. Run the scenario
    result = await scenario.run(
        name="dinner recipe request",
        description="""
            It's saturday evening, the user is very hungry and tired,
            but have no money to order out, so they are looking for a recipe.
        """,
        agents=[
            RecipeAgent(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(criteria=[
                "Agent should not ask more than two follow-up questions",
                "Agent should generate a recipe",
                "Recipe should include a list of ingredients",
                "Recipe should include step-by-step cooking instructions",
                "Recipe should be vegetarian and not include any sort of meat",
            ])
        ],
        script=[
            scenario.user("quick recipe for dinner"),
            scenario.agent(),
            scenario.user(),
            scenario.agent(),
            scenario.judge(),
        ],
    )
 
    # 3. Assert the result
    assert result.success
 
# Example agent implementation using litellm
@scenario.cache()
def vegetarian_recipe_agent(messages) -> scenario.AgentReturnTypes:
    response = litellm.completion(
        model="openai/gpt-4.1",
        messages=[
            {
                "role": "system",
                "content": """
                    You are a vegetarian recipe agent.
                    Given the user request, ask AT MOST ONE follow-up question,
                    then provide a complete recipe. Keep your responses concise and focused.
                """,
            },
            *messages,
        ],
    )
    return response.choices[0].message

2. Set Up Your Environment

OPENAI_API_KEY=<your-api-key>
LANGWATCH_API_KEY=<your-api-key> # optional, for simulation reporting

3. Run Your Test

python
uv run pytest -s test_my_agent.py

Visualize Your Results Instantly!

Console Output

The output displays the conversation between the simulated user and the agent, followed by the judge's evaluation:

What Happened?

This example demonstrates:

  1. Agent Integration: You created an AgentAdapter[ts][py]
  2. Simulation: The userSimulatorAgent[ts][py] automatically generated realistic user messages based on the scenario description
  3. Evaluation: The judgeAgent[ts][py] evaluated the conversation against your criteria
  4. Caching: The @scenario.cache()[py] decorator (Python) made your agent calls deterministic for testing

Key Concepts

  • Scenarios: Test cases that describe the situation and expected behavior
  • Agent Under Test: Your agent being tested on the scenario
  • User Simulator Agent: Agent that simulates user messages to interact with the agent under test
  • Judge Agent: Agent that decides if the conversation should proceed or not, while evaluating it against your criteria
  • Script: Optional way to control the exact flow of conversation

Next Steps