Module scenario.config

Explore Scenario configuration modules to define simulation rules, agent behavior, and evaluation flows for agent testing.

This module provides all configuration classes for customizing the behavior of the Scenario testing framework, including model settings, scenario execution parameters, and LangWatch integration.

Classes

ModelConfig: Configuration for LLM model settings ScenarioConfig: Main configuration for scenario execution LangWatchSettings: Configuration for LangWatch API integration

Example

from scenario.config import ModelConfig, ScenarioConfig, LangWatchSettings

# Configure LLM model
model_config = ModelConfig(
    model="openai/gpt-4.1-mini",
    temperature=0.1
)

# Configure scenario execution
scenario_config = ScenarioConfig(
    default_model=model_config,
    max_turns=15,
    verbose=True
)

# Configure LangWatch integration
langwatch_settings = LangWatchSettings()  # Reads from environment

"""
Explore Scenario configuration modules to define simulation rules, agent behavior, and evaluation flows for agent testing.

This module provides all configuration classes for customizing the behavior
of the Scenario testing framework, including model settings, scenario execution
parameters, and LangWatch integration.

Classes:
    ModelConfig: Configuration for LLM model settings
    ScenarioConfig: Main configuration for scenario execution
    LangWatchSettings: Configuration for LangWatch API integration

Example:
    ```
    from scenario.config import ModelConfig, ScenarioConfig, LangWatchSettings

    # Configure LLM model
    model_config = ModelConfig(
        model="openai/gpt-4.1-mini",
        temperature=0.1
    )

    # Configure scenario execution
    scenario_config = ScenarioConfig(
        default_model=model_config,
        max_turns=15,
        verbose=True
    )

    # Configure LangWatch integration
    langwatch_settings = LangWatchSettings()  # Reads from environment
    ```
"""

from .model import ModelConfig
from .scenario import ScenarioConfig
from .langwatch import LangWatchSettings

__all__ = [
    "ModelConfig",
    "ScenarioConfig",
    "LangWatchSettings",
]

Sub-modules

scenario.config.langwatch

Use LangWatch configuration in Scenario to link agent simulations with observability, evaluations, and AI agent testing …

scenario.config.logging

Logging configuration for Scenario …

scenario.config.model

Configure model settings in Scenario to define underlying LLM behavior for AI agent testing environments …

scenario.config.scenario

Access Scenario configuration in Python to define evaluation policies and structured agent testing behavior …

scenario.config.voice_models

Default model identifiers for voice paths …

Classes

class LangWatchSettings (**values: Any)

Configuration for LangWatch API integration.

This class handles configuration for connecting to LangWatch services, automatically reading from environment variables with the LANGWATCH_ prefix.

Attributes

endpoint

LangWatch API endpoint URL

api_key

API key for LangWatch authentication

project_id

Optional LangWatch project ID. When set, requests scope to this project via the X-Project-Id header. Required for API keys that are not bound to a single project.

Environment Variables: LANGWATCH_ENDPOINT: LangWatch API endpoint (defaults to https://app.langwatch.ai) LANGWATCH_API_KEY: API key for authentication (defaults to empty string) LANGWATCH_PROJECT_ID: Project ID for project-scoped requests (defaults to empty string)

Example

# Using environment variables
# export LANGWATCH_ENDPOINT="https://app.langwatch.ai"
# export LANGWATCH_API_KEY="your-api-key"
# export LANGWATCH_PROJECT_ID="project_xxx"

settings = LangWatchSettings()
print(settings.endpoint)    # <https://app.langwatch.ai>
print(settings.api_key)     # your-api-key
print(settings.project_id)  # project_xxx

# Or override programmatically
settings = LangWatchSettings(
    endpoint="https://custom.langwatch.ai",
    api_key="your-api-key",
    project_id="project_xxx",
)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Expand source code

class LangWatchSettings(BaseSettings):
    """
    Configuration for LangWatch API integration.

    This class handles configuration for connecting to LangWatch services,
    automatically reading from environment variables with the LANGWATCH_ prefix.

    Attributes:
        endpoint: LangWatch API endpoint URL
        api_key: API key for LangWatch authentication
        project_id: Optional LangWatch project ID. When set, requests scope to
            this project via the X-Project-Id header. Required for API keys
            that are not bound to a single project.

    Environment Variables:
        LANGWATCH_ENDPOINT: LangWatch API endpoint (defaults to https://app.langwatch.ai)
        LANGWATCH_API_KEY: API key for authentication (defaults to empty string)
        LANGWATCH_PROJECT_ID: Project ID for project-scoped requests (defaults to empty string)

    Example:
        ```
        # Using environment variables
        # export LANGWATCH_ENDPOINT="https://app.langwatch.ai"
        # export LANGWATCH_API_KEY="your-api-key"
        # export LANGWATCH_PROJECT_ID="project_xxx"

        settings = LangWatchSettings()
        print(settings.endpoint)    # https://app.langwatch.ai
        print(settings.api_key)     # your-api-key
        print(settings.project_id)  # project_xxx

        # Or override programmatically
        settings = LangWatchSettings(
            endpoint="https://custom.langwatch.ai",
            api_key="your-api-key",
            project_id="project_xxx",
        )
        ```
    """

    model_config = SettingsConfigDict(env_prefix="LANGWATCH_", case_sensitive=False)

    endpoint: HttpUrl = Field(
        default=HttpUrl("https://app.langwatch.ai"),
        description="LangWatch API endpoint URL",
    )
    api_key: str = Field(default="", description="API key for LangWatch authentication")
    project_id: str = Field(
        default="",
        description="LangWatch project ID, sent as X-Project-Id when present",
    )

Ancestors

• pydantic_settings.main.BaseSettings
• pydantic.main.BaseModel

Class variables

var api_key : str

var endpoint : pydantic.networks.HttpUrl

var model_config : ClassVar[pydantic_settings.main.SettingsConfigDict]

var project_id : str

class ModelConfig (**data: Any)

Configuration for LLM model settings.

This class encapsulates all the parameters needed to configure an LLM model for use with user simulator and judge agents in the Scenario framework.

The ModelConfig accepts any additional parameters that litellm supports, including headers, timeout, client, and other provider-specific options.

Attributes

model

The model identifier (e.g., "openai/gpt-4.1-mini", "anthropic/claude-3-sonnet")

api_base

Optional base URL where the model is hosted

api_key

Optional API key for the model provider

temperature

Sampling temperature for response generation (0.0 = deterministic, 1.0 = creative)

max_tokens

Maximum number of tokens to generate in responses

Example

# Basic configuration
model_config = ModelConfig(
    model="openai/gpt-4.1-mini",
    api_base="https://api.openai.com/v1",
    api_key="your-api-key",
    temperature=0.1,
    max_tokens=1000
)

# With custom headers and timeout
model_config = ModelConfig(
    model="openai/gpt-4",
    headers={"X-Custom-Header": "value"},
    timeout=60,
    num_retries=3
)

# With custom OpenAI client
from openai import OpenAI
model_config = ModelConfig(
    model="openai/gpt-4",
    client=OpenAI(
        base_url="https://custom.com",
        default_headers={"X-Auth": "token"}
    )
)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Expand source code

class ModelConfig(BaseModel):
    """
    Configuration for LLM model settings.

    This class encapsulates all the parameters needed to configure an LLM model
    for use with user simulator and judge agents in the Scenario framework.

    The ModelConfig accepts any additional parameters that litellm supports,
    including headers, timeout, client, and other provider-specific options.

    Attributes:
        model: The model identifier (e.g., "openai/gpt-4.1-mini", "anthropic/claude-3-sonnet")
        api_base: Optional base URL where the model is hosted
        api_key: Optional API key for the model provider
        temperature: Sampling temperature for response generation (0.0 = deterministic, 1.0 = creative)
        max_tokens: Maximum number of tokens to generate in responses

    Example:
        ```
        # Basic configuration
        model_config = ModelConfig(
            model="openai/gpt-4.1-mini",
            api_base="https://api.openai.com/v1",
            api_key="your-api-key",
            temperature=0.1,
            max_tokens=1000
        )

        # With custom headers and timeout
        model_config = ModelConfig(
            model="openai/gpt-4",
            headers={"X-Custom-Header": "value"},
            timeout=60,
            num_retries=3
        )

        # With custom OpenAI client
        from openai import OpenAI
        model_config = ModelConfig(
            model="openai/gpt-4",
            client=OpenAI(
                base_url="https://custom.com",
                default_headers={"X-Auth": "token"}
            )
        )
        ```
    """

    model_config = ConfigDict(extra="allow")

    model: str
    api_base: Optional[str] = None
    api_key: Optional[str] = None
    temperature: float = 0.0
    max_tokens: Optional[int] = None

Ancestors

• pydantic.main.BaseModel

Class variables

var api_base : str | None

var api_key : str | None

var max_tokens : int | None

var model : str

var model_config

var temperature : float

class ScenarioConfig (**data: Any)

Global configuration class for the Scenario testing framework.

This class allows users to set default behavior and parameters that apply to all scenario executions, including the LLM model to use for simulator and judge agents, execution limits, and debugging options.

Attributes

default_model

Default LLM model configuration for agents (can be string or ModelConfig)

max_turns

Maximum number of conversation turns before scenario times out

min_turns

Minimum number of turns that must run before the judge may volunteer a verdict (its finish_test tool is withheld on earlier turns). Explicit judge() steps and the final turn still deliver a terminal verdict. Must be a non-negative integer and must not exceed max_turns. Zero is valid. Unset by default.

verbose

Whether to show detailed output during execution (True/False or verbosity level)

cache_key

Key for caching scenario results to ensure deterministic behavior

debug

Whether to enable debug mode with step-by-step interaction

fetch_remote_traces

Whether the judge fetches the traces the agent under test reported to LangWatch for this conversation's trace ids and evaluates them alongside locally collected spans. Requires the agent adapter to forward AgentInput.propagation_headers to the remote agent. Off by default.

trace_wait_timeout

Maximum seconds the judge waits at verdict time for remote traces to arrive and stabilize, shared across all trace ids. Defaults to 30 seconds. Only used when fetch_remote_traces is enabled.

trace_wait_extension

Seconds for the judge's one extra wait. When the traces are still incomplete after the settle-wait, the verdict call offers a wait_for_traces tool: calling it waits this budget once more, then the tool is withdrawn and the judge must decide. Defaults to the resolved trace_wait_timeout.

observability

OpenTelemetry tracing configuration (span_filter, instrumentors, etc.)

Example

import scenario
from scenario import scenario_only

# Configure globally for all scenarios
scenario.configure(
    default_model="openai/gpt-4.1-mini",
    max_turns=15,
    observability={
        "span_filter": scenario_only,
        "instrumentors": [],
    },
)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Expand source code

class ScenarioConfig(BaseModel):
    """
    Global configuration class for the Scenario testing framework.

    This class allows users to set default behavior and parameters that apply
    to all scenario executions, including the LLM model to use for simulator
    and judge agents, execution limits, and debugging options.

    Attributes:
        default_model: Default LLM model configuration for agents (can be string or ModelConfig)
        max_turns: Maximum number of conversation turns before scenario times out
        min_turns: Minimum number of turns that must run before the judge may
            volunteer a verdict (its finish_test tool is withheld on earlier
            turns). Explicit judge() steps and the final turn still deliver a
            terminal verdict. Must be a non-negative integer and must not exceed
            max_turns. Zero is valid. Unset by default.
        verbose: Whether to show detailed output during execution (True/False or verbosity level)
        cache_key: Key for caching scenario results to ensure deterministic behavior
        debug: Whether to enable debug mode with step-by-step interaction
        fetch_remote_traces: Whether the judge fetches the traces the agent
            under test reported to LangWatch for this conversation's trace ids
            and evaluates them alongside locally collected spans. Requires the
            agent adapter to forward ``AgentInput.propagation_headers`` to the
            remote agent. Off by default.
        trace_wait_timeout: Maximum seconds the judge waits at verdict time
            for remote traces to arrive and stabilize, shared across all trace
            ids. Defaults to 30 seconds. Only used when ``fetch_remote_traces``
            is enabled.
        trace_wait_extension: Seconds for the judge's one extra wait. When
            the traces are still incomplete after the settle-wait, the
            verdict call offers a ``wait_for_traces`` tool: calling it waits
            this budget once more, then the tool is withdrawn and the judge
            must decide. Defaults to the resolved ``trace_wait_timeout``.
        observability: OpenTelemetry tracing configuration (span_filter, instrumentors, etc.)

    Example:
        ```
        import scenario
        from scenario import scenario_only

        # Configure globally for all scenarios
        scenario.configure(
            default_model="openai/gpt-4.1-mini",
            max_turns=15,
            observability={
                "span_filter": scenario_only,
                "instrumentors": [],
            },
        )
        ```
    """

    model_config = ConfigDict(arbitrary_types_allowed=True)

    default_model: Optional[Union[str, ModelConfig]] = None
    max_turns: Optional[int] = 10
    min_turns: Optional[int] = Field(default=None, strict=True, ge=0)
    verbose: Optional[Union[bool, int]] = True
    cache_key: Optional[str] = None
    debug: Optional[bool] = False
    headless: Optional[bool] = os.getenv("SCENARIO_HEADLESS", "false").lower() not in [
        "false",
        "0",
        "",
    ]
    fetch_remote_traces: Optional[bool] = None
    # allow_inf_nan=False on both: gt=0 alone accepts positive infinity, and
    # an infinite budget becomes a deadline the settle loop can never reach.
    trace_wait_timeout: Optional[float] = Field(
        default=None, gt=0, allow_inf_nan=False
    )
    trace_wait_extension: Optional[float] = Field(
        default=None, gt=0, allow_inf_nan=False
    )
    observability: Optional[Dict[str, Any]] = None

    default_config: ClassVar[Optional["ScenarioConfig"]] = None

    @classmethod
    def configure(
        cls,
        default_model: Optional[Union[str, ModelConfig]] = None,
        max_turns: Optional[int] = None,
        min_turns: Optional[int] = None,
        verbose: Optional[Union[bool, int]] = None,
        cache_key: Optional[str] = None,
        debug: Optional[bool] = None,
        headless: Optional[bool] = None,
        fetch_remote_traces: Optional[bool] = None,
        trace_wait_timeout: Optional[float] = None,
        trace_wait_extension: Optional[float] = None,
        observability: Optional[Dict[str, Any]] = None,
    ) -> None:
        """
        Set global configuration settings for all scenario executions.

        This method allows you to configure default behavior that will be applied
        to all scenarios unless explicitly overridden in individual scenario runs.

        Args:
            default_model: Default LLM model identifier for user simulator and judge agents
            max_turns: Maximum number of conversation turns before timeout (default: 10)
            min_turns: Minimum turns guaranteed before the judge may volunteer
                a verdict (unset by default; must be a non-negative integer no
                greater than max_turns; zero is valid)
            verbose: Enable verbose output during scenario execution
            cache_key: Cache key for deterministic scenario behavior across runs
            debug: Enable debug mode for step-by-step execution with user intervention
            fetch_remote_traces: Have the judge fetch the traces the agent
                under test reported to LangWatch for this conversation and
                evaluate them alongside locally collected spans (default: False).
                Requires the agent adapter to forward
                ``AgentInput.propagation_headers`` to the remote agent.
            trace_wait_timeout: Maximum seconds the judge waits at verdict
                time for remote traces to arrive and stabilize (default: 30)
            trace_wait_extension: Seconds for the judge's one extra wait via
                the ``wait_for_traces`` tool (default: the resolved
                ``trace_wait_timeout``)
            observability: OpenTelemetry tracing configuration. Accepts:
                - span_filter: Callable filter (use scenario_only or with_custom_scopes())
                - span_processors: List of additional SpanProcessors
                - trace_exporter: Custom SpanExporter
                - instrumentors: List of OTel instrumentors (pass [] to disable auto-instrumentation)

        Example:
            ```
            import scenario
            from scenario import scenario_only

            scenario.configure(
                default_model="openai/gpt-4.1-mini",
                observability={
                    "span_filter": scenario_only,
                    "instrumentors": [],
                },
            )

            # All subsequent scenario runs will use these defaults
            result = await scenario.run(
                name="my test",
                description="Test scenario",
                agents=[my_agent, scenario.UserSimulatorAgent(), scenario.JudgeAgent()]
            )
            ```
        """
        existing_config = cls.default_config or ScenarioConfig()

        cls.default_config = existing_config.merge(
            ScenarioConfig(
                default_model=default_model,
                max_turns=max_turns,
                min_turns=min_turns,
                verbose=verbose,
                cache_key=cache_key,
                debug=debug,
                headless=headless,
                fetch_remote_traces=fetch_remote_traces,
                trace_wait_timeout=trace_wait_timeout,
                trace_wait_extension=trace_wait_extension,
                observability=observability,
            )
        )

    def merge(self, other: "ScenarioConfig") -> "ScenarioConfig":
        """
        Merge this configuration with another configuration.

        Values from the other configuration will override values in this
        configuration where they are not None.

        Args:
            other: Another ScenarioConfig instance to merge with

        Returns:
            A new ScenarioConfig instance with merged values

        Example:
            ```
            base_config = ScenarioConfig(max_turns=10, verbose=True)
            override_config = ScenarioConfig(max_turns=20)

            merged = base_config.merge(override_config)
            # Result: max_turns=20, verbose=True
            ```
        """
        return ScenarioConfig(
            **{
                **self.items(),
                **other.items(),
            }
        )

    def items(self):
        """
        Get configuration items as a dictionary.

        Returns:
            Dictionary of configuration key-value pairs, excluding None values

        Example:
            ```
            config = ScenarioConfig(max_turns=15, verbose=True)
            items = config.items()
            # Result: {"max_turns": 15, "verbose": True}
            ```
        """
        return {k: getattr(self, k) for k in self.model_dump(exclude_none=True).keys()}

Ancestors

• pydantic.main.BaseModel

Class variables

var cache_key : str | None

var debug : bool | None

var default_config : ClassVar[ScenarioConfig | None]

var default_model : str | ModelConfig | None

var fetch_remote_traces : bool | None

var headless : bool | None

var max_turns : int | None

var min_turns : int | None

var model_config

var observability : Dict[str, Any] | None

var trace_wait_extension : float | None

var trace_wait_timeout : float | None

var verbose : bool | int | None

Static methods

def configure(default_model: str | ModelConfig | None = None, max_turns: int | None = None, min_turns: int | None = None, verbose: bool | int | None = None, cache_key: str | None = None, debug: bool | None = None, headless: bool | None = None, fetch_remote_traces: bool | None = None, trace_wait_timeout: float | None = None, trace_wait_extension: float | None = None, observability: Dict[str, Any] | None = None) ‑> None

Set global configuration settings for all scenario executions.

This method allows you to configure default behavior that will be applied to all scenarios unless explicitly overridden in individual scenario runs.

Args

default_model

Default LLM model identifier for user simulator and judge agents

max_turns

Maximum number of conversation turns before timeout (default: 10)

min_turns

Minimum turns guaranteed before the judge may volunteer a verdict (unset by default; must be a non-negative integer no greater than max_turns; zero is valid)

verbose

Enable verbose output during scenario execution

cache_key

Cache key for deterministic scenario behavior across runs

debug

Enable debug mode for step-by-step execution with user intervention

fetch_remote_traces

Have the judge fetch the traces the agent under test reported to LangWatch for this conversation and evaluate them alongside locally collected spans (default: False). Requires the agent adapter to forward AgentInput.propagation_headers to the remote agent.

trace_wait_timeout

Maximum seconds the judge waits at verdict time for remote traces to arrive and stabilize (default: 30)

trace_wait_extension

Seconds for the judge's one extra wait via the wait_for_traces tool (default: the resolved trace_wait_timeout)

observability

OpenTelemetry tracing configuration. Accepts: - span_filter: Callable filter (use scenario_only or with_custom_scopes()) - span_processors: List of additional SpanProcessors - trace_exporter: Custom SpanExporter - instrumentors: List of OTel instrumentors (pass [] to disable auto-instrumentation)

Example

import scenario
from scenario import scenario_only

scenario.configure(
    default_model="openai/gpt-4.1-mini",
    observability={
        "span_filter": scenario_only,
        "instrumentors": [],
    },
)

# All subsequent scenario runs will use these defaults
result = await scenario.run(
    name="my test",
    description="Test scenario",
    agents=[my_agent, scenario.UserSimulatorAgent(), scenario.JudgeAgent()]
)

Methods

def items(self)

Get configuration items as a dictionary.

Returns

Dictionary of configuration key-value pairs, excluding None values

Example

config = ScenarioConfig(max_turns=15, verbose=True)
items = config.items()
# Result: {"max_turns": 15, "verbose": True}

Expand source code

def items(self):
    """
    Get configuration items as a dictionary.

    Returns:
        Dictionary of configuration key-value pairs, excluding None values

    Example:
        ```
        config = ScenarioConfig(max_turns=15, verbose=True)
        items = config.items()
        # Result: {"max_turns": 15, "verbose": True}
        ```
    """
    return {k: getattr(self, k) for k in self.model_dump(exclude_none=True).keys()}

def merge(self, other: ScenarioConfig) ‑> ScenarioConfig

Merge this configuration with another configuration.

Values from the other configuration will override values in this configuration where they are not None.

Args

other

Another ScenarioConfig instance to merge with

Returns

A new ScenarioConfig instance with merged values

Example

base_config = ScenarioConfig(max_turns=10, verbose=True)
override_config = ScenarioConfig(max_turns=20)

merged = base_config.merge(override_config)
# Result: max_turns=20, verbose=True

Expand source code

def merge(self, other: "ScenarioConfig") -> "ScenarioConfig":
    """
    Merge this configuration with another configuration.

    Values from the other configuration will override values in this
    configuration where they are not None.

    Args:
        other: Another ScenarioConfig instance to merge with

    Returns:
        A new ScenarioConfig instance with merged values

    Example:
        ```
        base_config = ScenarioConfig(max_turns=10, verbose=True)
        override_config = ScenarioConfig(max_turns=20)

        merged = base_config.merge(override_config)
        # Result: max_turns=20, verbose=True
        ```
    """
    return ScenarioConfig(
        **{
            **self.items(),
            **other.items(),
        }
    )