Roadmap

Closed in v0.1 (conceptual core)

• Scope · Session · Behavior · Actor · Action · Target · Content
• Target semantics normativized: meaning is determined by Action category
• Variables · Capture · Runtime bindings (dataset, CLI flag, env var)
• Evaluations — step-level and session-level (chain), including blocking: true
• Evaluator types: 7 step-level (exact_match, contains, regex, schema, tool_call, llm_judge, custom), 6 chain (sequence, eventually, never, count, within, variable_consistency), plus composition (all_of/any_of/none_of)
• Vocabulary — four action categories (Communication, Execution, Interaction, Delegation)
• Sequencing · Alternate flows as separate Sessions
• Fragments (include:) for composition — inline, single-file, expanded before resolution
• Tool interaction spec (calls, responds, with/with_only, tool_call evaluator)
• JSON Schema (schema/abs.schema.json) — complete and validated by the CLI at parse time
• TypeScript implementation: parser, runner, all evaluators, CLI, OpenAI/Claude/Gemini adapters, AI Evaluator adapter, table formatter, test suite
• Python implementation: parser, runner, all evaluators, CLI, AI Evaluator adapter, table formatter
• UI authoring tool (ui/): React + Material UI, drag & drop, property sheets, YAML export
• Website: Next.js + Tailwind + MDX docs site
• VSCode extension: visual editor + ▶ Run button
• Mock agent (tools/mock_agent.py) for local testing

Closed in v0.2

• Optional behaviors (optional, requires, matches_when), the expected evaluator, when expressions, after constraints, and batch optional resolution.

Closed in v0.3

• Reference-based deterministic evaluators: f1, bleu, rouge with the ground_truth common field.
• Per-rule adapter: selection enforced in both implementations (named adapter registry).
• Structured logs and progress (--log-format jsonl, --log-level, --log-file, --no-log-content) and machine-readable code/details on evaluation results.
• Azure adapter on npm (local, prompt-based, no Python SDK); safety dimensions registered to the built-in judge in TypeScript.
• AWS Bedrock and Google Vertex AI adapters on npm.
• blocking failure propagation to inconclusive downstream results.

Remaining open design questions

• Vocabulary extensibility. Is action a closed enum or an open, extensible namespace? VOCABULARY.md currently recommends plain lowercase verbs as a stopgap; a formal extension syntax is still undecided.
• Actor/Action compatibility. Should the spec enforce which actors can perform which actions, or leave it to implementer-defined validation layers?
• Session composition — Background: style. Fragments already solve prefix deduplication, but a Gherkin-style Background: that auto-includes hasn't been designed.
• Parallel and retried behavior. v0.1 assumes a strictly linear sequence. Real agents issue parallel tool calls and retries.
• Content typing. content mixes free text, structured payloads, and "displayed" UI content under one untyped field.
• llm_judge calibration/reproducibility across model versions.
• Chain-evaluator selector — whether it needs partial/regex matching on target.
• Cross-Session variable sharing — explicitly out of scope for v0.1.
• Parameterized fragments — "same fragment, different values."
• Cross-file fragments — shared fragment libraries across repositories.

Implementation status

How to contribute

ABS is a draft standard. Issues and proposals should reference which document and section they affect. Open questions are tracked as GitHub issues.

View open issues →

Version timeline