CLI Reference

The CLI is the first thing anyone touches. It has to make three people happy at the same time:

• QA: "I want to run this against staging and see if it passed or not, now."
• Data: "I have 200 cases in a JSONL. Give me an aggregated report."
• Everyone: "I don't want to learn YAML. Just let me describe the behavior and generate the file for me."

All three should succeed on their first try.

The five commands

abslang init                    # Scaffold a project
abslang chat                    # Generate a session by describing it in plain language
abslang run                     # Execute sessions against an agent
abslang report                  # View results from a previous run
abslang generate-ci             # Generate a CI/CD workflow (GitHub Actions / GitLab CI)

abslang init

Creates a project skeleton in the current directory:

abslang init

Creates:

.
├── abs.config.yaml              # Project-level config (gitignored)
├── sessions/
│   └── order-status.abs.yaml    # Example session with {{placeholders}}
└── datasets/
    └── order-status.jsonl       # 3 rows that bind those placeholders

abslang chat

You don't need to know the YAML format. Describe what the agent should do in plain language and abslang chat builds the .abs.yaml for you:

abslang chat

🤖 ABS Assistant — describe the agent behavior you want to test

You: A customer asks for a refund on a damaged item.
     The agent should verify the order, process the refund,
     and confirm with the amount and reference.

Assistant: I'll draft a refund flow with tool calls…
           [generates complete .abs.yaml with evaluations,
            dataset placeholders, and chain checks]

Uses OPENAI_API_KEY, ANTHROPIC_API_KEY, or DEEPSEEK_API_KEY — whichever is set. Override the model with ABS_CHAT_MODEL and the base URL with ABS_CHAT_BASE_URL.

abslang run

Executes one or more Sessions against an agent.

Simplest invocation

abslang run sessions/order-status.abs.yaml --agent http://localhost:8080/chat

With a single variable override

abslang run sessions/order-status.abs.yaml --agent $URL --var orderId=12345

With a dataset

abslang run sessions/order-status.abs.yaml --agent $URL --dataset datasets/order-status.jsonl

Runs the session once per row in the dataset. The report aggregates everything.

With a dataset and a filter

abslang run sessions/order-status.abs.yaml --agent $URL --dataset datasets/order-status.jsonl --filter "orderId:12345"

Multiple sessions at once

abslang run sessions/ --agent $URL --dataset datasets/

Runs every .abs.yaml in the directory against every .jsonl whose filename starts with the same prefix.

All flags

Environment variables

Every flag can also be set via environment variable:

Structured logs and progress

run writes progress and events to stderr and the final report to stdout, so abslang run ... --format json > report.json stays clean:

abslang run session.abs.yaml --agent $URL --log-format jsonl --log-file events.jsonl

• --log-format pretty (default) or jsonl (one JSON object per line).
• --log-level error | warn | info | debug.
• --log-file always writes JSONL, regardless of the console format.
• --no-log-content omits trace content and reasons (privacy).

Events cover run, session, agent.request/response/error, behavior.match/skipped, evaluation.result, and report.written, with a stable envelope (v, ts, level, event, run_id). Failed evaluations also carry a machine-readable code (evaluator.threshold_not_met, adapter.not_configured, …) alongside the human reason.

Evaluator adapters

Pick where LLM-based evaluations are judged — without touching the session file:

# Built-in judge — auto-detects OpenAI / Anthropic / Gemini
OPENAI_API_KEY=sk-... abslang run session.abs.yaml --agent $URL

# Built-in judge on any OpenAI-compatible endpoint (Azure/Foundry, Ollama, vLLM, gateway)
abslang run session.abs.yaml --agent $URL \
  --judge-base-url "https://<resource>.openai.azure.com/openai/v1" \
  --judge-api-key "$AZURE_OPENAI_API_KEY" \
  --judge-api-key-header api-key \
  --judge-model gpt-4o-mini

# Azure AI Foundry — quality dimensions (Groundedness, Relevance, …)
abslang run session.abs.yaml --agent $URL --adapter azure

# AWS Bedrock — llm_judge + custom metrics
abslang run session.abs.yaml --agent $URL --adapter aws

# Google Vertex AI — quality + safety
abslang run session.abs.yaml --agent $URL --adapter google

# AI Evaluator — free tier, no infrastructure
abslang run session.abs.yaml --agent $URL --adapter llm_judge=aievaluator

Safety dimensions (Violence, HateUnfairness, Sexual, SelfHarm) work with the built-in judge out of the box — no criteria required. See Evaluations.

azure ships with both the Python and npm CLIs (the npm adapter renders the official Azure prompt templates locally, without the Python SDK). aws and google also ship on npm — they need @aws-sdk/client-bedrock-runtime and @google-cloud/vertexai respectively.

abslang report

Reads a JSON report from a previous abslang run --output and displays it.

abslang run session.abs.yaml --agent $URL --output report.json
abslang report report.json

abslang report report.json --format table   # Default, human-readable
abslang report report.json --format json    # Machine-readable
abslang report report.json --format junit   # CI integration
abslang report report.json --failed         # Show only failed cases
abslang report report.json --detail 3       # Show full trace for case #3

Config file

Every option can be set three ways: as a CLI flag, as an ABS_* environment variable, or in a config file. Precedence:

CLI flag  >  environment variable  >  config file  >  built-in default

By default ABS loads abs.config.yaml (searched from the current directory upward), or an explicit file with --config-file path.yaml (YAML or JSON). Unknown sections/keys are reported as warnings so typos don't silently get ignored.

# abs.config.yaml — every key maps to a CLI flag / ABS_* env var
agent:
  url: http://localhost:8080/chat   # --agent / ABS_AGENT_URL
  format: openai                    # --agent-format / ABS_AGENT_FORMAT
  auth: none                        # --agent-auth / ABS_AGENT_AUTH
  token: sk-...                     # --agent-token / ABS_AGENT_TOKEN
  model: gpt-4o                     # --agent-model / ABS_AGENT_MODEL
  forward_auth: true                # --agent-forward-auth / ABS_AGENT_FORWARD_AUTH
  authorization: "Bearer eyJ..."    # --agent-authorization / ABS_AGENT_AUTHORIZATION
  refresh_url: ...                  # --agent-refresh-url / ABS_AGENT_REFRESH_URL
  refresh_token: ...                # --agent-refresh-token / ABS_AGENT_REFRESH_TOKEN
  client_id: ...                    # --agent-client-id / ABS_AGENT_CLIENT_ID

adapters:
  llm_judge: aievaluator            # --adapter llm_judge=aievaluator

judge:
  base_url: ...                     # --judge-base-url / ABS_JUDGE_BASE_URL
  api_key: ...                      # --judge-api-key / ABS_JUDGE_API_KEY
  api_key_header: ...               # --judge-api-key-header / ABS_JUDGE_API_KEY_HEADER
  model: ...                        # --judge-model / ABS_JUDGE_MODEL

defaults:
  dataset: datasets/                # --dataset / ABS_DATASET
  filter: key:value                 # --filter / ABS_FILTER
  timeout: 120                      # --timeout / ABS_TIMEOUT
  parallel: 4                       # --parallel / ABS_PARALLEL
  output: report.json               # --output / ABS_OUTPUT
  vars:                             # fixed bindings (ABS_VAR_* / --var override)
    orderId: "12345"

run:
  format: table                     # --format / ABS_FORMAT (table | json | junit)
  ci: true                          # --ci / ABS_CI

logging:
  format: pretty                    # --log-format / ABS_LOG_FORMAT
  level: info                       # --log-level / ABS_LOG_LEVEL
  file: run.log.jsonl               # --log-file / ABS_LOG_FILE
  no_content: true                  # --no-log-content / ABS_NO_LOG_CONTENT

report:
  format: table                     # --format / ABS_FORMAT
  failed: false                     # --failed / ABS_FAILED
  detail: 3                         # --detail / ABS_DETAIL

CI/CD integration

abslang run sessions/ --agent $STAGING_AGENT --dataset datasets/ --format junit --ci > report.xml

Exit code 0 if all sessions passed. Exit code 1 if any failed. Drops straight into any CI system.

The four personas the CLI serves

One file. Four use cases.