Minute 15–20: sequence, variable_consistency — properties that span the entire trace.
So far we've evaluated individual steps. But there are things you can only verify by looking at the entire conversation:
These go in an evaluations: block at the same level as behaviors:, not inside a step:
evaluations:
# 1. The 3 assistant actions MUST happen in this relative order
- type: sequence
order:
- { actor: assistant, action: clarifies }
- { actor: assistant, action: informs }
- { actor: assistant, action: confirms }
# 2. The refund ID must be consistent everywhere
- type: variable_consistency
variable: refundId
What each one does:
sequence: verifies the three assistant actions happen in that relative order. It doesn't matter if there are other steps in between — it only matters that clarifies is before informs is before confirms. If the agent jumps straight to the resolution without clarifying first, the sequence fails.
variable_consistency: remember capture: refundId: "R-5512" from turn 2? This checks that every time a refund ID appears anywhere in the trace, it's the same value. Catches subtle bugs where the agent says "R-5512" early and "R-5513" later.
| Evaluator | What it verifies |
|---|---|
sequence | Steps in relative order |
eventually | Something happens at least once |
never | Something never happens (e.g. never hands off to human) |
count | Something happens exactly N times |
within | A happens within N steps after B |
variable_consistency | A captured value stays the same |
About 60 lines. PO understands the conversation. Dev knows what to build. QA has 7 automated checks across 3 stages. One file.
Next: Run it →