CLI Reference

The CLI is the first thing anyone touches. It has to make two people happy at the same time:

• QA: "I want to run this against staging and see if it passed or not, now."
• Data: "I have 200 cases in a JSONL. Give me an aggregated report."

Both should succeed on their first try.

The three commands

abslang init                    # Scaffold a project
abslang run                     # Execute sessions against an agent
abslang report                  # View results from a previous run

abslang init

Creates a project skeleton in the current directory:

abs init

Creates:

.
├── abs.config.yaml              # Project-level config (gitignored)
├── sessions/
│   └── order-status.abs.yaml    # Example session with {{placeholders}}
└── datasets/
    └── order-status.jsonl       # 3 rows that bind those placeholders

abslang run

Executes one or more Sessions against an agent.

Simplest invocation

abslang run sessions/order-status.abs.yaml --agent http://localhost:8080/chat

With a single variable override

abslang run sessions/order-status.abs.yaml --agent $URL --var orderId=12345

With a dataset

abslang run sessions/order-status.abs.yaml --agent $URL --dataset datasets/order-status.jsonl

Runs the session once per row in the dataset. The report aggregates everything.

With a dataset and a filter

abslang run sessions/order-status.abs.yaml --agent $URL --dataset datasets/order-status.jsonl --filter "orderId:12345"

Multiple sessions at once

abslang run sessions/ --agent $URL --dataset datasets/

Runs every .abs.yaml in the directory against every .jsonl whose filename starts with the same prefix.

All flags

Environment variables

Every flag can also be set via environment variable:

abslang report

Reads a JSON report from a previous abslang run --output and displays it.

abslang run session.abs.yaml --agent $URL --output report.json
abslang report report.json

abslang report report.json --format table   # Default, human-readable
abslang report report.json --format json    # Machine-readable
abslang report report.json --format junit   # CI integration
abslang report report.json --failed         # Show only failed cases
abslang report report.json --detail 3       # Show full trace for case #3

Config file

abs.config.yaml stores project-level settings:

agent:
  url: http://localhost:8080/chat
  format: openai
  auth: none

adapters:
  llm_judge: aievaluator

defaults:
  dataset: datasets/
  timeout: 120

Precedence: CLI flag > environment variable > config file > built-in default.

CI/CD integration

abslang run sessions/ --agent $STAGING_AGENT --dataset datasets/ --format junit --ci > report.xml

Exit code 0 if all sessions passed. Exit code 1 if any failed. Drops straight into any CI system.

The four personas the CLI serves

One file. Four use cases.