3. Step-level evaluations

Describing is fine, but you're QA: you want to verify the bot actually did everything right. That's what evaluations are for.

There are two critical moments in this flow: when the bot asks for verification (was it empathetic? did it ask for the right info?), and when it delivers the final result (are all the facts there? did it hallucinate anything?).

Let's add evaluations at those two points:

  # Step 2 improved: we verify response quality
  - actor: assistant
    action: asks
    content: "I'm sorry about the damage. Can you confirm your name and order date?"
    evaluations:
      - type: llm_judge
        criteria: |
          1. Shows empathy for the damaged item
          2. Mentions order number #8291
          3. Asks for verification before taking action

llm_judge asks an AI to evaluate the response against your criteria, in natural language. Zero code.

For hard facts — things that absolutely must appear — we use exact verifications:

  # Step 8 improved: hard facts + quality
  - actor: assistant
    action: informs
    content: "Refund of €47.50 processed, Franco. You'll receive it in 3-5 days. Reference: R-5512."
    capture:
      refundId: "R-5512"
    evaluations:
      # Hard fact: the ID must appear
      - type: contains
        value: "R-5512"

      # Quality: tone, completeness
      - type: llm_judge
        criteria: |
          1. States the amount (€47.50) and timeline (3-5 days)
          2. Gives the reference R-5512
          3. Uses the customer's name (Franco)
          4. Reassuring tone, no upsells or deflections

capture stores the value R-5512 for later use. You'll need it in the next section.

Available evaluation types

Built-in (no adapter needed, runs locally):

LLM-based (auto-detects OpenAI/Anthropic/Gemini from env, or route through an adapter):

What backs llm_judge and dimension evaluators

When you add an llm_judge, Groundedness, Relevance, Coherence, or Fluency evaluation, someone has to call an LLM to produce the judgment. abslang gives you multiple paths — including keeping everything local for privacy — and you don't have to change your session file to switch between them.

Evaluator types: what runs where

Built-in judge: zero setup for llm_judge

Out of the box, abslang auto-detects whichever LLM provider you have available:

# OpenAI — if OPENAI_API_KEY is set
OPENAI_API_KEY=sk-... abslang run session.abs.yaml --agent $URL

# Anthropic — if ANTHROPIC_API_KEY is set
ANTHROPIC_API_KEY=sk-ant-... abslang run session.abs.yaml --agent $URL

# Gemini — if GEMINI_API_KEY is set
GEMINI_API_KEY=... abslang run session.abs.yaml --agent $URL

Set ABS_JUDGE_PROVIDER if you have more than one and want to pick:

ABS_JUDGE_PROVIDER=openai abslang run session.abs.yaml --agent $URL

Override the model with ABS_JUDGE_MODEL:

ABS_JUDGE_MODEL=gpt-4o-mini abslang run session.abs.yaml --agent $URL

No API key? Use the mock judge:

ABS_MOCK_JUDGE=true abslang run session.abs.yaml --agent $URL

Dimension evaluators: Groundedness, Relevance, Coherence, Fluency

These measure specific quality dimensions. They require an evaluator adapter — the built-in judge only handles llm_judge.

With AI Evaluator (recommended):

# 100 free evals/month with API key
AIEVALUATOR_API_KEY=... abslang run session.abs.yaml \
  --agent $URL \
  --adapter llm_judge=aievaluator

# 5 free evals/day without API key (playground)
abslang run session.abs.yaml \
  --agent $URL \
  --adapter llm_judge=aievaluator

With your own LLM (full privacy, no external API calls):

Deploy a private judge behind any OpenAI-compatible endpoint (Ollama, vLLM, LiteLLM):

# Local Ollama
abslang run session.abs.yaml \
  --agent $URL \
  --adapter llm_judge=local \
  --adapter-url http://localhost:11434/v1

# Self-hosted vLLM
abslang run session.abs.yaml \
  --agent $URL \
  --adapter llm_judge=local \
  --adapter-url https://judge.internal.company.com/v1 \
  --adapter-key $JUDGE_API_KEY

Nothing leaves your network.

With Azure AI / Vertex AI:

abslang run session.abs.yaml \
  --agent $URL \
  --adapter llm_judge=azure \
  --adapter-key $AZURE_KEY \
  --adapter-url $AZURE_ENDPOINT

Config file — don't retype adapter flags

# abs.config.yaml
adapters:
  llm_judge: aievaluator       # Route all LLM evaluations through AI Evaluator
  Groundedness: aievaluator    # Or split by type
  Relevance: local              # Use local judge for relevance

The adapter contract

Any adapter is a function that receives (trace, evaluationRule) and returns { passed, score, reason }. Providers (Azure, LangSmith, Promptfoo, Galileo, Arize) can ship adapters implementing this interface. Your session file stays the same — only the --adapter flag changes.

adapter.evaluate(
  trace = [...conversation steps...],
  rule  = { type: "Groundedness", query: ..., context: ..., response: ..., threshold: 0.8 }
) → { passed: true, score: 0.92, reason: "..." }

Next: Chain evaluations →