Skip to content

Blackbox Testing

Blackbox testing evaluates software functionality through external interfaces without examining internal implementation. Learn more about blackbox testing.

For Scenario Users

Scenario blackbox testing requires adapters to call the agent's public interface—the same HTTP endpoint, CLI command, or SDK method that users interact with—rather than directly instantiating agent classes or mocking internal components.

This tests your agent as it runs in production: with real databases, authentication, middleware, and infrastructure.

Example

typescript
import scenario, { AgentAdapter, AgentRole } from "@langwatch/scenario";
 
// ✅ Blackbox: adapter calls real HTTP endpoint (no mocking)
const adapter: AgentAdapter = {
  role: AgentRole.AGENT,
  call: async (input) => {
    // Make real HTTP request to actual agent server
    return await fetch("http://localhost:3000/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ messages: input.messages }),
    })
      .then((r) => r.json())
      .then((r) => r.response);
  },
};
 
// Scenario tests the complete production stack
await scenario.run({
  agents: [scenario.userSimulatorAgent(), adapter],
  // ... scenario configuration
});

Compare with mocked testing where you might mock tool functions or bypass HTTP entirely. Blackbox tests the complete stack.

Judging Internal Behavior via Remote Traces

The endpoint returns final text only, so a plain blackbox judge can only grade the words: a criterion like "the agent looked the order up before answering" is unverifiable from the response. Remote traces close that gap. The adapter forwards the scenario's trace context to your endpoint, your agent reports its spans to LangWatch under the same trace, and the judge fetches those spans and verifies internal behavior against them, so blackbox tests keep the production stack and still judge tool calls, writes, and retrievals on real evidence.

Enable it on the scenario, and forward input.propagation_headers (Python) / input.propagationHeaders (TypeScript) in your adapter:

typescript
await scenario.run({
  // ...
  fetchRemoteTraces: true,
});

See Remote Traces for the full setup, including the server-side trace context adoption.

Use Cases

  • Production validation: Verify deployed agent works correctly
  • Integration testing: Test all components together (server, database, agent)
  • API contracts: Ensure public interface doesn't break

See Also