Evaluations

ABS does not have two modes with two syntaxes. A document with no evaluations anywhere is a pure behavioral description. The same document, with evaluations added, becomes executable as an automated test. Nothing about the document's shape changes — evaluations is simply optional annotation.

Two levels

Step-level evaluations attach to a single Behavior and check that one observed step is correct in isolation.

Session-level (chain) evaluations attach to the Session as a whole and check properties of the entire observed trace: ordering, consistency, invariants.

session: Refund request — approved
behaviors:
  - actor: user
    action: says
    content: "I want to return order #8291, it arrived damaged"

  - actor: assistant
    action: asks
    content: "I'm sorry about that. Can you confirm your name and order date?"
    evaluations:
      - type: llm_judge
        criteria: |
          1. Shows empathy for the damaged item
          2. References the order number #8291
          3. Asks for verification info before taking action

  - actor: user
    action: says
    content: "Franco Vinciarelli, ordered last Tuesday"
    capture:
      customerName: "Franco Vinciarelli"

  - actor: assistant
    action: calls
    target: Orders API
  - actor: tool
    action: responds
    target: Orders API
  - actor: assistant
    action: calls
    target: Refunds API
  - actor: tool
    action: responds
    target: Refunds API

  - actor: assistant
    action: informs
    content: "Refund of €47.50 processed, Franco. Refund ID: R-5512."
    capture:
      refundId: "R-5512"
    evaluations:
      - type: contains
        value: "R-5512"
      - type: llm_judge
        criteria: |
          1. States refund amount and timeline
          2. Provides refund reference
          3. Uses customer's name
          4. Reassuring tone, no upsells

evaluations:
  - type: sequence
    order:
      - { actor: assistant, action: asks }
      - { actor: assistant, action: calls, target: "Orders API" }
      - { actor: assistant, action: calls, target: "Refunds API" }
      - { actor: assistant, action: informs }
  - type: variable_consistency
    variable: refundId
  - type: never
    match: { actor: assistant, action: hands_off }

Step-level evaluator types

exact_match

The observed content must equal value exactly.

evaluations:
  - type: exact_match
    value: "12345"

contains

The observed content must contain value as a substring (case-insensitive by default).

evaluations:
  - type: contains
    value: "on the way"

regex

The observed content must match the given pattern.

evaluations:
  - type: regex
    pattern: "^Order #\\d+ is (on the way|delivered)$"

schema

The observed content must validate against a JSON Schema.

evaluations:
  - type: schema
    schema:
      type: object
      required: [orderId]
      properties:
        orderId: { type: string }

tool_call

Validates that one or more calls Behaviors were invoked correctly: target matches, parameters match (per with/with_only rules), and optionally that ordering is respected when multiple tools are called. See Tools & MCP for the full rules.

evaluations:
  - type: tool_call
    target: "Order MCP"
    with:
      orderId: "{{orderId}}"
    ordered: true

llm_judge

Delegates the judgment to an LLM against a natural-language rubric.

evaluations:
  - type: llm_judge
    criteria: "Response clearly states the order is in transit, in a friendly tone, without inventing a delivery date."

custom

Escape hatch for evaluators not covered above. Implementations define their own id namespace.

evaluations:
  - type: custom
    id: my-org.sentiment-positive

Session-level (chain) evaluator types

All chain evaluators use a selector to identify Behaviors in the trace: an object with any of actor, action, target. A field that's present must match exactly; a field omitted is a wildcard.

match: { actor: assistant, action: calls, target: "Order MCP" }

sequence

Every selector in order must match some Behavior in the trace, in increasing position (not necessarily adjacent — other Behaviors may fall between them).

- type: sequence
  order:
    - { actor: assistant, action: calls, target: "Order MCP" }
    - { actor: assistant, action: informs }

eventually

The selector must match at least one Behavior somewhere in the trace.

- type: eventually
  match: { actor: assistant, action: informs }

never

The selector must match no Behavior anywhere in the trace.

- type: never
  match: { actor: assistant, action: hands_off }

count

Bounds how many Behaviors match the selector.

- type: count
  match: { actor: assistant, action: calls }
  min: 1
  max: 2

within

A match selector must occur within max_steps Behaviors after an after selector.

- type: within
  after: { actor: user, action: says }
  match: { actor: assistant, action: responds }
  max_steps: 3

variable_consistency

Every value that resolves a given {{variable}} reference anywhere in the Session — plus the value it was originally captured with — must be equal. Catches an agent silently substituting a different order number or ID than the one the user actually gave.

- type: variable_consistency
  variable: orderId

Composition: all_of / any_of / none_of

Available at both levels. Wraps a list of nested evaluations:

- type: any_of
  evaluations:
    - type: contains
      value: "on the way"
    - type: contains
      value: "in transit"

Listing multiple evaluations directly under evaluations: (without wrapping) is already an implicit all_of. The wrapper types exist specifically for any_of/none_of, which a flat list cannot express.

Failure semantics

Default: non-blocking / best-effort. A failing evaluation does not stop other evaluations from being checked. An implementation MUST evaluate every evaluations entry it can and produce a full pass/fail report.

blocking: true. An evaluation MAY set blocking: true as a checkpoint. If a blocking: true evaluation fails, any other evaluation whose Behavior depends — directly or via a captured variable — on the step that failed SHOULD be reported as inconclusive rather than failed, avoiding a wall of misleading downstream failures.

- actor: assistant
  action: calls
  target: Order MCP
  with:
    orderId: "{{orderId}}"
  evaluations:
    - type: tool_call
      target: "Order MCP"
      blocking: true

The Session's overall result is pass if and only if every evaluation in the report passed.