CLI Reference

The CLI is the first thing anyone touches. It has to make three people happy at the same time:

• QA: "I want to run this against staging and see if it passed or not, now."
• Data: "I have 200 cases in a JSONL. Give me an aggregated report."
• Everyone: "I don't want to learn YAML. Just let me describe the behavior and generate the file for me."

All three should succeed on their first try.

The five commands

abslang init                    # Scaffold a project
abslang chat                    # Generate a session by describing it in plain language
abslang run                     # Execute sessions against an agent
abslang report                  # View results from a previous run
abslang generate-ci             # Generate a CI/CD workflow (GitHub Actions / GitLab CI)

abslang init

Creates a project skeleton in the current directory:

abslang init

Creates:

.
├── abs.config.yaml              # Project-level config (gitignored)
├── sessions/
│   └── order-status.abs.yaml    # Example session with {{placeholders}}
└── datasets/
    └── order-status.jsonl       # 3 rows that bind those placeholders

abslang chat

You don't need to know the YAML format. Describe what the agent should do in plain language and abslang chat builds the .abs.yaml for you:

abslang chat

🤖 ABS Assistant — describe the agent behavior you want to test

You: A customer asks for a refund on a damaged item.
     The agent should verify the order, process the refund,
     and confirm with the amount and reference.

Assistant: I'll draft a refund flow with tool calls…
           [generates complete .abs.yaml with evaluations,
            dataset placeholders, and chain checks]

Uses OPENAI_API_KEY, ANTHROPIC_API_KEY, or DEEPSEEK_API_KEY — whichever is set. Override the model with ABS_CHAT_MODEL and the base URL with ABS_CHAT_BASE_URL.

abslang run

Executes one or more Sessions against an agent.

Simplest invocation

abslang run sessions/order-status.abs.yaml --agent http://localhost:8080/chat

With a single variable override

abslang run sessions/order-status.abs.yaml --agent $URL --var orderId=12345

With a dataset

abslang run sessions/order-status.abs.yaml --agent $URL --dataset datasets/order-status.jsonl

Runs the session once per row in the dataset. The report aggregates everything.

With a dataset and a filter

abslang run sessions/order-status.abs.yaml --agent $URL --dataset datasets/order-status.jsonl --filter "orderId:12345"

Multiple sessions at once

abslang run sessions/ --agent $URL --dataset datasets/

Runs every .abs.yaml in the directory against every .jsonl whose filename starts with the same prefix.

All flags

Environment variables

Every flag can also be set via environment variable:

Structured logs and progress

run writes progress and events to stderr and the final report to stdout, so abslang run ... --format json > report.json stays clean:

abslang run session.abs.yaml --agent $URL --log-format jsonl --log-file events.jsonl

• --log-format pretty (default) or jsonl (one JSON object per line).
• --log-level error | warn | info | debug.
• --log-file always writes JSONL, regardless of the console format.
• --no-log-content omits trace content and reasons (privacy).

Events cover run, session, agent.request/response/error, behavior.match/skipped, evaluation.result, and report.written, with a stable envelope (v, ts, level, event, run_id). Failed evaluations also carry a machine-readable code (evaluator.threshold_not_met, adapter.not_configured, …) alongside the human reason.

Evaluator adapters

Pick where LLM-based evaluations are judged — without touching the session file:

# Built-in judge — auto-detects OpenAI / Anthropic / Gemini
OPENAI_API_KEY=sk-... abslang run session.abs.yaml --agent $URL

# Built-in judge on any OpenAI-compatible endpoint (Azure/Foundry, Ollama, vLLM, gateway)
abslang run session.abs.yaml --agent $URL \
  --judge-base-url "https://<resource>.openai.azure.com/openai/v1" \
  --judge-api-key "$AZURE_OPENAI_API_KEY" \
  --judge-api-key-header api-key \
  --judge-model gpt-4o-mini

# Azure AI Foundry — quality dimensions (Groundedness, Relevance, …)
abslang run session.abs.yaml --agent $URL --adapter azure

# AWS Bedrock — llm_judge + custom metrics
abslang run session.abs.yaml --agent $URL --adapter aws

# Google Vertex AI — quality + safety
abslang run session.abs.yaml --agent $URL --adapter google

# AI Evaluator — free tier, no infrastructure
abslang run session.abs.yaml --agent $URL --adapter llm_judge=aievaluator

Safety dimensions (Violence, HateUnfairness, Sexual, SelfHarm) work with the built-in judge out of the box — no criteria required. See Evaluations.

azure ships with both the Python and npm CLIs (the npm adapter renders the official Azure prompt templates locally, without the Python SDK). aws and google also ship on npm — they need @aws-sdk/client-bedrock-runtime and @google-cloud/vertexai respectively.

abslang report

Reads a JSON report from a previous abslang run --output and displays it.

abslang run session.abs.yaml --agent $URL --output report.json
abslang report report.json

abslang report report.json --format table   # Default, human-readable
abslang report report.json --format json    # Machine-readable
abslang report report.json --format junit   # CI integration
abslang report report.json --failed         # Show only failed cases
abslang report report.json --detail 3       # Show full trace for case #3

Config file

abs.config.yaml stores project-level settings:

agent:
  url: http://localhost:8080/chat
  format: openai
  auth: none

adapters:
  llm_judge: aievaluator

defaults:
  dataset: datasets/
  timeout: 120

Precedence: CLI flag > environment variable > config file > built-in default.

CI/CD integration

abslang run sessions/ --agent $STAGING_AGENT --dataset datasets/ --format junit --ci > report.xml

Exit code 0 if all sessions passed. Exit code 1 if any failed. Drops straight into any CI system.

The four personas the CLI serves

One file. Four use cases.