Runner

The Runner takes an ABS session, plays it against a real agent, and tells you whether the agent behaved as expected. It is the piece that makes ABS not just documentation, but a test.

The two things you need

To run an ABS session you only need two things:

1. A session file — your .abs.yaml
2. An agent URL — an HTTP endpoint that speaks a simple chat protocol

abslang run ./order-status.abs.yaml --agent http://localhost:8080/chat

Architecture

1. Parse the ABS document into a sequence of expected Behaviors.
2. Execute by driving the agent — sending user messages, observing responses, detecting tool calls.
3. Capture the actual trace (what really happened).
4. Evaluate by comparing expected vs. actual, running all evaluations.
5. Report the results.

The Agent Contract

For the Runner to talk to an agent, the agent needs one HTTP endpoint:

Request

POST /chat
Content-Type: application/json

{
  "messages": [
    { "role": "user", "content": "Where is my order?" }
  ]
}

Response (text)

{
  "message": {
    "role": "assistant",
    "content": "Please provide your order number"
  }
}

Response (tool call)

{
  "message": {
    "role": "assistant",
    "content": null,
    "tool_calls": [
      {
        "id": "call_1",
        "type": "function",
        "function": {
          "name": "get_order",
          "arguments": "{\"orderId\": \"12345\"}"
        }
      }
    ]
  }
}

The Runner will send the tool result back and continue. This is the same shape used by OpenAI, Anthropic, and most open-source models.

Agent adapters

Agents speak different protocols. The Runner ships with built-in adapters:

abslang run session.abs.yaml --agent $URL --agent-format openai
abslang run session.abs.yaml --agent $URL --agent-format claude
abslang run session.abs.yaml --agent $URL --agent-format gemini

If your agent speaks a different protocol, write a thin adapter implementing send(messages) → response.

Authentication

How the Runner thinks

The Runner has one simple rule:

Everything with actor: user goes to the agent. Everything else is what the agent should do in response.

Session (what you wrote)          Reality (what the agent did)
═══════════════════════           ═══════════════════════════

user says "Hi"          ──────→   POST /chat
assistant greets        ←──────   { "role": "assistant", "content": "Hello!" }
                                    ✅  matched

user says "Order 12345" ──────→   POST /chat
assistant calls          ←──────   { "tool_calls": [{"function": {"name": "Order MCP"}}] }
  Order MCP                           ✅  matched

tool responds            ←──────   { "role": "tool", "content": "{\"status\": \"shipped\"}" }
                                    ✅  matched

assistant informs        ←──────   { "role": "assistant", "content": "Your order has shipped!" }
  "Your order..."                    ✅  matched
     │
     └── evaluation: contains "shipped"  →  ✅  PASS

Execution model

1. Parse the session YAML
2. Expand fragments into a flat list of Behaviors
3. Create an empty trace
4. For each Behavior, in order:
   a. If actor is "user":
      - Send the content to the agent
      - Collect the agent's response
      - Append both to the trace
   b. If actor is "assistant" and action is "calls":
      - Match the agent's tool_calls against target and with
      - If a tool response follows, send it back to continue
   c. If actor is "tool":
      - Skip — handled as the tool response payload in (b)
      - Its evaluations still run against the observed response
   d. If actor is anything else (assistant says/informs/asks/etc):
      - Match the agent's response against this Behavior
5. Run all step-level evaluations against the trace
6. Run all session-level (chain) evaluations against the full trace
7. Print the report
8. Exit 0 if everything passed, exit 1 otherwise

Matching rules

• actor and action must match exactly (communication actions are equivalent: says/asks/informs/etc. all match each other)
• target (if present) must match the observed tool name, recipient, or UI element
• with (if present) must match the observed tool arguments — partial by default, strict with with_only
• content is checked for structural compatibility

Evaluator adapters

ABS defines what to check. An evaluator adapter defines how to check it. The built-in evaluators (exact_match, contains, regex, schema, tool_call, sequence, eventually, never, count, within, variable_consistency) ship with the Runner. llm_judge, Groundedness, Relevance, Coherence, Fluency, and custom go through adapters.

The default adapter for llm_judge is AI Evaluator. Other providers (Azure AI, LangSmith, Promptfoo) can ship adapters implementing the same interface.

abslang run session.abs.yaml \
  --adapter llm_judge=azure \
  --adapter llm_judge=langsmith

Report formats

Table (default, human)

┌──────────────────────────────────────────────────────────┐
│  ABS — Results                                           │
├──────────────────────────────────────────────────────────┤
│  Session:  Order status requires order number            │
│  Agent:    http://localhost:8080/chat                     │
│  Result:   ✅ PASSED                                     │
│  Steps:    5/5 matched · 4/4 evaluations passed          │
├────┬────────────────────────────────────┬────────┬───────┤
│  # │ Step                               │ Result │       │
├────┼────────────────────────────────────┼────────┼───────┤
│  1 │ user says "Where is my order?"     │   →    │  sent │
│  2 │ assistant asks "order number"      │   ✅   │ match │
│  3 │ user says "12345"                  │   →    │  sent │
│  4 │ assistant calls Order MCP          │   ✅   │ match │
│  5 │ assistant informs "on the way"     │   ✅   │ match │
│    │   └─ contains "on the way"         │   ✅   │  pass │
│  C │ sequence: asks → calls → informs   │   ✅   │  pass │
│  C │ variable_consistency: orderId      │   ✅   │  pass │
│  C │ never: hands_off                   │   ✅   │  pass │
└────┴────────────────────────────────────┴────────┴───────┘

JSON (--format json)

Machine-readable output for downstream tooling and dashboards.

JUnit XML (--format junit)

For CI pipelines (GitHub Actions, GitLab CI, Jenkins).

What the Runner does NOT do

The Runner is not an agent framework. It does not:

• Write prompts or orchestrate chains of thought
• Modify the agent under test
• Decide which model or framework the agent should use
• Require the agent to be built a certain way

It only plays the user's part, watches the agent, and reports what happened.

See also

• CLI Reference — full command-line reference
• Evaluations — all evaluator types and semantics
• Tools & MCP — tool call matching rules
• Variables — capture, reuse, and runtime binding