Interface ScenarioEvaluator

The evaluator to run: a built-in type such as ragas/sql_query_equivalence or a saved evaluator as evaluators/<slug>, the same reference the evaluate endpoint accepts.

Where each evaluator input reads from, keyed by input name: a function of the scenario state or a literal. An input without a mapping is inferred from its name; a tool call is never inferred.

Whether a failing verdict fails the scenario. Defaults to true for an evaluator that answers pass or fail, and to false for a score-only one. A score never fails the scenario, whatever this flag says.

Per-run overrides of the evaluator settings.