Evaluations

ABS does not have two modes with two syntaxes. A document with no evaluations anywhere is a pure behavioral description. The same document, with evaluations added, becomes executable as an automated test. Nothing about the document's shape changes — evaluations is simply optional annotation.

Two levels

Step-level evaluations attach to a single Behavior and check that one observed step is correct in isolation.

Session-level (chain) evaluations attach to the Session as a whole and check properties of the entire observed trace: ordering, consistency, invariants.

session: Refund request — approved
behaviors:
  - actor: user
    action: says
    content: "I want to return order #8291, it arrived damaged"

  - actor: assistant
    action: asks
    content: "I'm sorry about that. Can you confirm your name and order date?"
    evaluations:
      - type: llm_judge
        criteria: |
          1. Shows empathy for the damaged item
          2. References the order number #8291
          3. Asks for verification info before taking action

  - actor: user
    action: says
    content: "Franco Vinciarelli, ordered last Tuesday"
    capture:
      customerName: "Franco Vinciarelli"

  - actor: assistant
    action: calls
    target: Orders API
  - actor: tool
    action: responds
    target: Orders API
  - actor: assistant
    action: calls
    target: Refunds API
  - actor: tool
    action: responds
    target: Refunds API

  - actor: assistant
    action: informs
    content: "Refund of €47.50 processed, Franco. Refund ID: R-5512."
    capture:
      refundId: "R-5512"
    evaluations:
      - type: contains
        value: "R-5512"
      - type: llm_judge
        criteria: |
          1. States refund amount and timeline
          2. Provides refund reference
          3. Uses customer's name
          4. Reassuring tone, no upsells

evaluations:
  - type: sequence
    order:
      - { actor: assistant, action: asks }
      - { actor: assistant, action: calls, target: "Orders API" }
      - { actor: assistant, action: calls, target: "Refunds API" }
      - { actor: assistant, action: informs }
  - type: variable_consistency
    variable: refundId
  - type: never
    match: { actor: assistant, action: hands_off }

Step-level evaluator types

exact_match

The observed content must equal value exactly.

evaluations:
  - type: exact_match
    value: "12345"

contains

The observed content must contain value as a substring (case-insensitive by default).

evaluations:
  - type: contains
    value: "on the way"

regex

The observed content must match the given pattern.

evaluations:
  - type: regex
    pattern: "^Order #\\d+ is (on the way|delivered)$"

schema

The observed content must validate against a JSON Schema.

evaluations:
  - type: schema
    schema:
      type: object
      required: [orderId]
      properties:
        orderId: { type: string }

tool_call

Validates that one or more calls Behaviors were invoked correctly: target matches, parameters match (per with/with_only rules), and optionally that ordering is respected when multiple tools are called. See Tools & MCP for the full rules.

evaluations:
  - type: tool_call
    target: "Order MCP"
    with:
      orderId: "{{orderId}}"
    ordered: true

llm_judge

Delegates the judgment to an LLM against a natural-language rubric.

evaluations:
  - type: llm_judge
    criteria: "Response clearly states the order is in transit, in a friendly tone, without inventing a delivery date."

custom

Escape hatch for evaluators not covered above. Implementations define their own id namespace.

evaluations:
  - type: custom
    id: my-org.sentiment-positive

Reference-based evaluators — f1, bleu, rouge

Deterministic metrics against a declared reference. No model, no network, no cost — ideal for CI regression on expected answers:

evaluations:
  - type: f1
    ground_truth: self          # the behavior's declared content
    threshold: 0.6
  - type: rouge
    ground_truth: "{{cases.expectedAnswer}}"
    variant: rougeL
    metric: f1
    threshold: 0.5

f1 measures token-level overlap, bleu n-gram precision (BLEU-4 with add-1 smoothing), and rouge n-gram / longest-common-subsequence overlap (variant: rouge1 | rouge2 | rougeL, metric: precision | recall | f1). ground_truth resolves like query/context/response: self (the declared content of the behavior), a trace reference (kb.responds), or an already-resolved dataset value. Results expose the full precision/recall/f1 breakdown in details. For semantic similarity instead of token overlap, use llm_judge or a provider custom id such as azure.similarity.

Quality dimensions — Groundedness, Relevance, Coherence, Fluency

Standard quality checks every major platform supports. Point them at the trace and route through an adapter (--adapter azure, --adapter google, or AI Evaluator):

evaluations:
  - type: Groundedness
    query: user_asks.says
    context: kb_result.responds
    response: self
    threshold: 0.8

Inputs are behavior-id references (user_asks.says, kb_result.responds); self is the behavior carrying the evaluation. The Runner records the matched behavior's id on the trace step, so these refs resolve against the observed trace.

Safety dimensions — HateUnfairness, Violence, Sexual, SelfHarm

Judge the output for harmful content. Each ships with a curated rubric, so they run on the built-in judge out of the box — no criteria required:

evaluations:
  - type: Violence
    response: self
    threshold: 0.9

1.0 = safe, 0.0 = harmful. Override with criteria: for a stricter definition. Pick the engine per run with --adapter, or per rule with adapter:.

Session-level (chain) evaluator types

All chain evaluators use a selector to identify Behaviors in the trace: an object with any of actor, action, target. A field that's present must match exactly; a field omitted is a wildcard. The observed trace includes the user turns the Runner sent, so selectors like { actor: user, action: says } work as expected.

Text responses are recorded as responds unless a declared communication behavior (asks, informs, greets, …) matched them: the runner annotates the observed step with the action of the first behavior that matched it. So { actor: assistant, action: asks } only matches responses your session classified as asks. Give the behavior a matches_when criterion when the distinction must be precise.

match: { actor: assistant, action: calls, target: "Order MCP" }

sequence

Every selector in order must match some Behavior in the trace, in increasing position (not necessarily adjacent — other Behaviors may fall between them).

- type: sequence
  order:
    - { actor: assistant, action: calls, target: "Order MCP" }
    - { actor: assistant, action: informs }

eventually

The selector must match at least one Behavior somewhere in the trace.

- type: eventually
  match: { actor: assistant, action: informs }

never

The selector must match no Behavior anywhere in the trace.

- type: never
  match: { actor: assistant, action: hands_off }

count

Bounds how many Behaviors match the selector.

- type: count
  match: { actor: assistant, action: calls }
  min: 1
  max: 2

within

A match selector must occur within max_steps Behaviors after an after selector.

- type: within
  after: { actor: user, action: says }
  match: { actor: assistant, action: responds }
  max_steps: 3

variable_consistency

Every value that resolves a given {{variable}} reference anywhere in the Session — plus the value it was originally captured with — must be equal. Catches an agent silently substituting a different order number or ID than the one the user actually gave.

- type: variable_consistency
  variable: orderId

Composition: all_of / any_of / none_of

Available at both levels. Wraps a list of nested evaluations:

- type: any_of
  evaluations:
    - type: contains
      value: "on the way"
    - type: contains
      value: "in transit"

Listing multiple evaluations directly under evaluations: (without wrapping) is already an implicit all_of. The wrapper types exist specifically for any_of/none_of, which a flat list cannot express.

Failure semantics

Default: non-blocking / best-effort. A failing evaluation does not stop other evaluations from being checked. An implementation MUST evaluate every evaluations entry it can and produce a full pass/fail report.

blocking: true. An evaluation MAY set blocking: true as a checkpoint. If a blocking: true evaluation fails, any other evaluation whose Behavior depends — directly or via a captured variable — on the step that failed SHOULD be reported as inconclusive rather than failed, avoiding a wall of misleading downstream failures.

- actor: assistant
  action: calls
  target: Order MCP
  with:
    orderId: "{{orderId}}"
  evaluations:
    - type: tool_call
      target: "Order MCP"
      blocking: true

The Session's overall result is pass if and only if every evaluation in the report passed.