Skip to content

Remote Traces

Judge remote agents on what they did, not what they said

Remote trace fetching makes the judge download the traces the agent under test reported to LangWatch before delivering its verdict. The fetched spans merge into the same trace digest the judge already reads, so progressive discovery and the expand_trace / grep_trace tools work on them unchanged.

You need this when the agent runs behind an HTTP endpoint and returns final text only, the blackbox testing setup. The tool calls, database writes, and document lookups happen on the server, never reach the local span collector, and a criterion like "the agent queried the order system" is unverifiable from the transcript alone.

It takes two pieces: forward the trace context to your agent, and enable fetching on the scenario.

1. Forward the trace context

Every scenario turn opens one trace and stamps its id on the turn's messages. AgentInput exposes that turn's W3C trace context as ready-to-send headers (traceparent, plus tracestate when set). Spread them onto the outgoing request in your adapter:

python
class RemoteAgentAdapter(scenario.AgentAdapter):
    async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
        async with aiohttp.ClientSession() as session:
            async with session.post(
                "https://your-agent.com/api/chat",
                json={"message": input.last_new_user_message_str()},
                # Forward the turn's trace context to the remote agent
                headers=dict(input.propagation_headers),
            ) as response:
                result = await response.json()
                return result["response"]

On the server side, your agent adopts the incoming context so the spans it creates land in the same trace. With standard OpenTelemetry HTTP instrumentation this adoption is automatic and needs zero code. Without it, extract the context and start your root span under it:

python
from opentelemetry import propagate, trace
 
ctx = propagate.extract(dict(request.headers))
tracer = trace.get_tracer("my-agent")
with tracer.start_as_current_span("chat", context=ctx):
    ...  # your agent logic, spans created here join the scenario's trace

2. Enable remote fetching

Turn on fetching per run, or project-wide:

python
result = await scenario.run(
    name="order status",
    description="A customer asks where their order is",
    agents=[RemoteAgentAdapter(), scenario.UserSimulatorAgent(), scenario.JudgeAgent(criteria=[...])],
    fetch_remote_traces=True,
    trace_wait_timeout=45.0,  # seconds, default 30
)
 
# or as a global default:
scenario.configure(fetch_remote_traces=True, trace_wait_timeout=45.0)

At judgment time, the judge collects every distinct trace id stamped on the conversation's messages (one per turn, not only the last), fetches each from GET {LANGWATCH_ENDPOINT}/api/trace/{id}, filters out scenario infrastructure spans, deduplicates against locally collected spans, and feeds the rest into its trace digest.

When the judge waits

Trace ingestion takes a moment, and the fetch schedule is built so conversations never slow down for it:

  • Conversation turns never fetch. While the conversation runs, the judge only decides whether to continue or move to the verdict, from the transcript and local spans. Remote traces are not fetched mid-conversation and never delay a turn.
  • Every verdict settle-waits first. When the judge moves to its verdict (a make_verdict decision, the last turn, or an explicit judge step), it polls each pending trace every second until the trace is complete, meaning it holds at least one span from your agent and every fetched span's parent span has arrived. A failed poll retries until the deadline instead of failing the trace. All traces are polled in parallel under the single shared trace_wait_timeout / traceWaitTimeoutMs budget.
  • A timeout never hides what did arrive. When the budget runs out on an incomplete trace, the judge still sees every span that was fetched, plus a synthetic error span that marks the trace as possibly incomplete. Criteria the visible spans prove can still pass; criteria that need the missing spans go inconclusive.
  • The judge can wait once more. When the traces are still incomplete at the verdict, the judge gets a one-shot wait_for_traces tool. It calls the tool only when the missing spans are essential for the verdict; the wait re-arms the failed traces and settle-waits one extra period, set by trace_wait_extension / traceWaitExtensionMs (default: the wait budget itself). After that wait the tool is withdrawn and the judge must deliver its verdict on the evidence at hand.
  • An inconclusive verdict continues the conversation, unless the evidence is dead. A voluntary verdict that comes back inconclusive sends the conversation onward for more turns. When no trace of the run ever settled, the inconclusive verdict is terminal instead: more turns cannot produce the missing evidence, and another settle-wait would only repeat the same timeout.

When trace collection fails

On timeout or fetch failure, the judge does not silently proceed. A synthetic span named langwatch.span_collection.error carrying the failure reason is added to the digest, and the reason decides how strict the judge is. When no agent spans arrived at all, criteria that depend on internal behavior go inconclusive, never passed from the transcript alone. When the trace arrived but is still incomplete at the deadline, the spans that are present still count: criteria they prove may pass, and only criteria whose evidence is missing stay inconclusive. See How Judging Works for the exact rule.

Requirements

  • LANGWATCH_API_KEY must be set (and LANGWATCH_ENDPOINT when not using LangWatch Cloud); the trace fetch authenticates with the same credentials the scenario uses for reporting.
  • The agent under test must report its traces to the same LangWatch project the scenario runs with. If it reports to a different project (or not at all), the fetch finds nothing and trace-dependent criteria come back inconclusive.

Full example

A test server stands in for the deployed agent: it adopts the propagated trace context, calls a tool, and answers with final text only. The judge passes the trace criterion because it sees the tool span, not because the response mentions it.

python
import aiohttp
import pytest
from aiohttp import web
from opentelemetry import propagate
from opentelemetry import trace as otel_trace
 
import scenario
 
base_url = ""
 
 
class RemoteAgentAdapter(scenario.AgentAdapter):
    async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
        async with aiohttp.ClientSession() as session:
            async with session.post(
                f"{base_url}/chat",
                json={"message": input.last_new_user_message_str()},
                headers=dict(input.propagation_headers),
            ) as response:
                result = await response.json()
                return result["response"]
 
 
async def chat_handler(request: web.Request) -> web.Response:
    await request.json()
 
    # Adopt the incoming trace context: spans created here join the
    # scenario turn's trace.
    context = propagate.extract(dict(request.headers))
    tracer = otel_trace.get_tracer("remote-weather-agent")
    with tracer.start_as_current_span("chat_request", context=context):
        with tracer.start_as_current_span("get_weather") as tool_span:
            tool_span.set_attribute("tool.name", "get_weather")
            weather = "The weather in London is sunny, 22 degrees Celsius."
            tool_span.set_attribute("tool.result", weather)
 
    # Final text only: the tool call above is never mentioned to the caller.
    return web.json_response({"response": weather})
 
 
@pytest.mark.asyncio
async def test_remote_trace_propagation():
    global base_url
    app = web.Application()
    app.router.add_post("/chat", chat_handler)
    runner = web.AppRunner(app)
    await runner.setup()
    site = web.TCPSite(runner, "localhost", 0)
    await site.start()
    port = site._server.sockets[0].getsockname()[1]
    base_url = f"http://localhost:{port}"
 
    try:
        result = await scenario.run(
            name="Remote trace propagation",
            description="User asks for the weather in London",
            agents=[
                RemoteAgentAdapter(),
                scenario.UserSimulatorAgent(model="openai/gpt-5-mini"),
                scenario.JudgeAgent(
                    model="openai/gpt-5-mini",
                    criteria=[
                        "Agent tells the user the weather in London",
                        "The traces show the agent called the get_weather tool",
                    ],
                ),
            ],
            script=[
                scenario.user("What's the weather like in London?"),
                scenario.agent(),
                scenario.judge(),
            ],
            fetch_remote_traces=True,
            trace_wait_timeout=20.0,
        )
        assert result.success
    finally:
        await runner.cleanup()

Next Steps