Runner

The Runner takes an ABS session, plays it against a real agent, and tells you whether the agent behaved as expected. It is the piece that makes ABS not just documentation, but a test.

The two things you need

To run an ABS session you only need two things:

1. A session file — your .abs.yaml
2. An agent URL — an HTTP endpoint that speaks a simple chat protocol

abslang run ./order-status.abs.yaml --agent http://localhost:8080/chat

Architecture

1. Parse the ABS document into a sequence of expected Behaviors.
2. Execute by driving the agent — sending user messages, observing responses, detecting tool calls.
3. Capture the actual trace (what really happened).
4. Evaluate by comparing expected vs. actual, running all evaluations.
5. Report the results.

The Agent Contract

For the Runner to talk to an agent, the agent needs one HTTP endpoint:

Request

POST /chat
Content-Type: application/json

{
  "messages": [
    { "role": "user", "content": "Where is my order?" }
  ]
}

Response (text)

{
  "message": {
    "role": "assistant",
    "content": "Please provide your order number"
  }
}

Response (tool call)

{
  "message": {
    "role": "assistant",
    "content": null,
    "tool_calls": [
      {
        "id": "call_1",
        "type": "function",
        "function": {
          "name": "get_order",
          "arguments": "{\"orderId\": \"12345\"}"
        }
      }
    ]
  }
}

The Runner will send the tool result back and continue. This is the same shape used by OpenAI, Anthropic, and most open-source models.

Agent adapters

Agents speak different protocols. The Runner ships with built-in adapters:

abslang run session.abs.yaml --agent $URL --agent-format openai
abslang run session.abs.yaml --agent $URL --agent-format responses  # Responses API + SSE
abslang run session.abs.yaml --agent $URL --agent-format claude
abslang run session.abs.yaml --agent $URL --agent-format gemini

With responses, the Runner sends model (--agent-model, default gpt-4o), translates the conversation to the Responses input format, and streams by default, folding response.output_text.delta, response.function_call_arguments.* and response.completed events into the trace.

If your agent speaks a different protocol, write a thin adapter implementing send(messages) → response.

Authentication

How the Runner thinks

The Runner has one simple rule:

Everything with actor: user goes to the agent. Everything else is what the agent should do in response.

Session (what you wrote)          Reality (what the agent did)
═══════════════════════           ═══════════════════════════

user says "Hi"          ──────→   POST /chat
assistant greets        ←──────   { "role": "assistant", "content": "Hello!" }
                                    ✅  matched

user says "Order 12345" ──────→   POST /chat
assistant calls          ←──────   { "tool_calls": [{"function": {"name": "Order MCP"}}] }
  Order MCP                           ✅  matched

tool responds            ←──────   { "role": "tool", "content": "{\"status\": \"shipped\"}" }
                                    ✅  matched

assistant informs        ←──────   { "role": "assistant", "content": "Your order has shipped!" }
  "Your order..."                    ✅  matched
     │
     └── evaluation: contains "shipped"  →  ✅  PASS

Execution model

1. Parse the session YAML
2. Expand fragments into a flat list of Behaviors
3. Create an empty trace
4. For each Behavior, in order:
   a. If actor is "user":
      - Send the content to the agent
      - Collect the agent's response
      - Append both to the trace
   b. If actor is "assistant" and action is "calls":
      - Match the agent's tool_calls against target and with
      - Append the tool call to the trace
   c. If actor is "tool":
      - Send the content back as the tool response
      - Append the tool response and the agent's continuation to the trace
      - Its evaluations still run against the observed response
   d. If actor is anything else (assistant says/informs/asks/etc):
      - Match the agent's response against this Behavior
5. Run all step-level evaluations against the trace
6. Run all session-level (chain) evaluations against the full trace
7. Print the report
8. Exit 0 if everything passed, exit 1 otherwise

Matching rules

• actor and action must match exactly (communication actions are equivalent: says/asks/informs/etc. all match each other)
• When a behavior matches a text response, the observed step is annotated with that behavior's action (first match wins). Chain-evaluation selectors (never, sequence, count, within, eventually) compare exactly, so never: { action: asks } only fires when an asks behavior matched the response
• target (if present) must match the observed tool name, recipient, or UI element
• with (if present) must match the observed tool arguments — partial by default, strict with with_only
• content is checked for structural compatibility

Evaluator adapters

ABS defines what to check. An evaluator adapter defines how to check it. The built-in evaluators (exact_match, contains, regex, schema, tool_call, sequence, eventually, never, count, within, variable_consistency) ship with the Runner. llm_judge, Groundedness, Relevance, Coherence, Fluency, and custom go through adapters.

The default adapter for llm_judge is the built-in judge — it auto-detects OpenAI, Anthropic, or Gemini from your environment. Azure AI Foundry, AWS Bedrock, and Google Vertex AI ship as ready-to-use engines; safety dimensions (HateUnfairness, Violence, Sexual, SelfHarm) run on the built-in judge with a curated rubric — no criteria.

# Built-in judge
OPENAI_API_KEY=sk-... abslang run session.abs.yaml --agent $URL

# Built-in judge on any OpenAI-compatible endpoint (Azure/Foundry, Ollama, vLLM, gateway)
abslang run session.abs.yaml --agent $URL \
  --judge-base-url "https://<resource>.openai.azure.com/openai/v1" \
  --judge-api-key "$AZURE_OPENAI_API_KEY" \
  --judge-api-key-header api-key \
  --judge-model gpt-4o-mini

# Azure AI Foundry
abslang run session.abs.yaml --agent $URL --adapter azure

# AWS Bedrock
abslang run session.abs.yaml --agent $URL --adapter aws

# Google Vertex AI
abslang run session.abs.yaml --agent $URL --adapter google

# AI Evaluator
abslang run session.abs.yaml --agent $URL --adapter llm_judge=aievaluator

Report formats

Table (default, human)

┌──────────────────────────────────────────────────────────┐
│  ABS — Results                                           │
├──────────────────────────────────────────────────────────┤
│  Session:  Order status requires order number            │
│  Agent:    http://localhost:8080/chat                     │
│  Result:   ✅ PASSED                                     │
│  Steps:    5/5 matched · 4/4 evaluations passed          │
├────┬────────────────────────────────────┬────────┬───────┤
│  # │ Step                               │ Result │       │
├────┼────────────────────────────────────┼────────┼───────┤
│  1 │ user says "Where is my order?"     │   →    │  sent │
│  2 │ assistant asks "order number"      │   ✅   │ match │
│  3 │ user says "12345"                  │   →    │  sent │
│  4 │ assistant calls Order MCP          │   ✅   │ match │
│  5 │ assistant informs "on the way"     │   ✅   │ match │
│    │   └─ contains "on the way"         │   ✅   │  pass │
│  C │ sequence: asks → calls → informs   │   ✅   │  pass │
│  C │ variable_consistency: orderId      │   ✅   │  pass │
│  C │ never: hands_off                   │   ✅   │  pass │
└────┴────────────────────────────────────┴────────┴───────┘

JSON (--format json)

Machine-readable output for downstream tooling and dashboards.

JUnit XML (--format junit)

For CI pipelines (GitHub Actions, GitLab CI, Jenkins).

What the Runner does NOT do

The Runner is not an agent framework. It does not:

• Write prompts or orchestrate chains of thought
• Modify the agent under test
• Decide which model or framework the agent should use
• Require the agent to be built a certain way

It only plays the user's part, watches the agent, and reports what happened.

See also

• CLI Reference — full command-line reference
• Evaluations — all evaluator types and semantics
• Tools & MCP — tool call matching rules
• Variables — capture, reuse, and runtime binding