3. Step-level evaluations

Describing is fine, but you're QA: you want to verify the agent did everything right at every stage. That's what evaluations are for — and the real power of ABS is that you can place them at different points in the same flow, each one seeing the trace accumulated so far.

Turn 1: did the agent understand the intent?

  - actor: assistant
    action: clarifies
    content: "I understand your order #8291 arrived damaged. I'll help you get a refund."
    evaluations:
      - type: llm_judge
        criteria: |
          1. Correctly classifies the intent as a refund request
          2. References the order number #8291
          3. Acknowledges the damage (not a simple return)
          4. Takes ownership of the resolution

llm_judge asks an AI to evaluate the response against your criteria in natural language. Zero code. At this point the evaluator only sees the clarification — the trace has just one assistant response so far.

Turn 2: did it deliver the facts completely?

  - actor: assistant
    action: informs
    content: "Refund of €47.50 approved. Reference: R-5512. You'll receive it in 3-5 days."
    capture:
      refundId: "R-5512"
    evaluations:
      # Hard fact: the ID must appear
      - type: contains
        value: "R-5512"

      # Soft qualities: tone, completeness
      - type: llm_judge
        criteria: |
          1. States the exact refund amount (€47.50)
          2. Provides the reference number R-5512
          3. Sets a clear timeline (3-5 days)
          4. Professional and empathetic tone

      # Did it answer what was asked?
      - type: Relevance
        query: user
        response: self

Three evaluators on a single step: contains for the hard fact, llm_judge for quality, Relevance to verify the answer matches the question. The evaluator now sees the clarification plus the resolution — the trace has grown.

capture stores the refund ID for later. You'll need it in the next section.

Turn 3: is the closing appropriate?

  - actor: assistant
    action: confirms
    content: "You're welcome! Is there anything else I can help with?"
    evaluations:
      - type: llm_judge
        criteria: |
          1. Offers further assistance
          2. Does NOT reopen the resolved refund
          3. Concise and natural

By now the evaluator sees the entire conversation: clarification + resolution + closing. Evaluations get richer as the flow progresses.

This is what makes ABS different from a single-shot eval tool: five evaluations, across three stages, each with a growing view of the trace. The same session file, the same adapter, different slices of context.

What backs llm_judge and the dimension evaluators

When you use llm_judge, Relevance, Groundedness, Coherence, Fluency, or a safety dimension, an LLM produces the judgment. You pick where — and the session file never changes.

# Built-in judge — auto-detects OpenAI, Anthropic, or Gemini from env
OPENAI_API_KEY=sk-... abslang run session.abs.yaml --agent $URL

# Azure AI Foundry — quality dimensions + agentic evaluators
abslang run session.abs.yaml --agent $URL --adapter azure

# AWS Bedrock — llm_judge + custom metrics
abslang run session.abs.yaml --agent $URL --adapter aws

# Google Vertex AI — quality + safety
abslang run session.abs.yaml --agent $URL --adapter google

# AI Evaluator — free tier
abslang run session.abs.yaml --agent $URL --adapter llm_judge=aievaluator

Safety dimensions — no criteria to write

Some checks are so common you shouldn't have to write the rubric. Safety dimensions ship with a curated one:

  - actor: assistant
    action: informs
    evaluations:
      - type: Violence
        threshold: 0.9
      - type: HateUnfairness
        threshold: 0.9

1.0 = safe, 0.0 = harmful. They run on the built-in judge out of the box. Override with criteria: when you need stricter. See the full adapter guide.

Next: Chain evaluations →