6. Datasets and golden data

So far we've run a single case with hardcoded data. But with AI agents you need more: what about a broken item? A wrong item? A cracked screen? Each with its own expected response and evaluation criteria.

ABS answers this with datasets. Turn your session into a template with {{placeholders}} and run it against a data file.

Step 1: Replace fixed values with placeholders

Instead of hardcoding every message, use {{cases.column}}:

session: Damaged item → refund (parametrized)
dataset:
  id: cases
  path: refund-cases.jsonl
behaviors:
  # ── Turn 1 ──
  - actor: user
    action: says
    content: "{{cases.userMessage}}"

  - actor: assistant
    action: clarifies
    content: "{{cases.expectedClarification}}"
    evaluations:
      - type: llm_judge
        criteria: "{{cases.clarificationCriteria}}"

  # ── Turn 2 ──
  - actor: user
    action: says
    content: "{{cases.followUp}}"

  - actor: assistant
    action: informs
    content: "{{cases.expectedResolution}}"
    capture:
      refundId: "{{cases.expectedRefundId}}"
    evaluations:
      - type: contains
        value: "{{cases.expectedRefundId}}"
      - type: llm_judge
        criteria: "{{cases.resolutionCriteria}}"
      - type: Relevance
        query: user
        response: self

  # ── Turn 3 ──
  - actor: user
    action: says
    content: "{{cases.closing}}"

  - actor: assistant
    action: confirms
    content: "{{cases.expectedClosing}}"
    evaluations:
      - type: llm_judge
        criteria: "{{cases.closingCriteria}}"

evaluations:
  - type: sequence
    order:
      - { actor: assistant, action: clarifies }
      - { actor: assistant, action: informs }
      - { actor: assistant, action: confirms }
  - type: variable_consistency
    variable: refundId

Everything can be parametrized: user messages, expected responses, even the evaluation criteria themselves. Each scenario can define its own quality bar.

Step 2: Create the dataset

A .jsonl file — one JSON object per row, one row per test scenario:

{"userMessage":"I received a damaged item, I want my money back. Order #8291.","expectedClarification":"I understand your order #8291 arrived damaged. I'll help you get a refund.","clarificationCriteria":"1. Classifies as refund\n2. References #8291\n3. Acknowledges damage\n4. Takes ownership","followUp":"Yes please, how long?","expectedResolution":"Refund of €47.50 approved. Reference: R-5512. 3-5 days.","expectedRefundId":"R-5512","resolutionCriteria":"1. States amount €47.50\n2. Reference R-5512\n3. Timeline 3-5 days\n4. Professional tone","closing":"Great, thanks.","expectedClosing":"You're welcome! Anything else?","closingCriteria":"1. Offers further help\n2. Doesn't reopen refund\n3. Concise"}
{"userMessage":"My order #3412 arrived broken, refund please.","expectedClarification":"Sorry about order #3412 arriving broken...","clarificationCriteria":"1. Classifies as refund\n2. References #3412\n3. Acknowledges breakage\n4. Takes ownership","followUp":"How fast is the refund?","expectedResolution":"Refund of €32.00 approved. Reference: R-7811. 5-7 business days.","expectedRefundId":"R-7811","resolutionCriteria":"1. States amount €32.00\n2. Reference R-7811\n3. Timeline 5-7 days\n4. Professional tone","closing":"Thanks!","expectedClosing":"You're welcome! Anything else I can do?","closingCriteria":"1. Offers further help\n2. Doesn't reopen refund\n3. Concise"}
{"userMessage":"Wrong item in my box for order #5567, I want a refund.","expectedClarification":"I see order #5567 had the wrong item...","clarificationCriteria":"1. Classifies as refund\n2. References #5567\n3. Acknowledges wrong item\n4. Takes ownership","followUp":"Yes, go ahead.","expectedResolution":"Refund of €89.99 approved. Reference: R-3394. 3-5 days.","expectedRefundId":"R-3394","resolutionCriteria":"1. States amount €89.99\n2. Reference R-3394\n3. Timeline 3-5 days\n4. Professional tone","closing":"Perfect, thanks.","expectedClosing":"Glad to help! Anything else?","closingCriteria":"1. Offers further help\n2. Doesn't reopen refund\n3. Concise"}
{"userMessage":"Order #1234 came with a cracked screen, refund please.","expectedClarification":"That's not acceptable — your order #1234 arrived with a cracked screen...","clarificationCriteria":"1. Classifies as refund\n2. References #1234\n3. Acknowledges cracked screen\n4. Takes ownership","followUp":"How many days?","expectedResolution":"Refund of €215.00 approved. Reference: R-9904. 3-5 days.","expectedRefundId":"R-9904","resolutionCriteria":"1. States amount €215.00\n2. Reference R-9904\n3. Timeline 3-5 days\n4. Professional tone","closing":"Awesome, thank you.","expectedClosing":"You're welcome! Anything else I can help with?","closingCriteria":"1. Offers further help\n2. Doesn't reopen refund\n3. Concise"}

Four scenarios — damaged, broken, wrong item, cracked screen — each with its own messages, expected responses, and evaluation criteria. Same session structure, different data.

Step 3: Run everything

abslang run refund-parametrized.abs.yaml \
  --agent $URL \
  --dataset refund-cases.jsonl

Four rows, four runs. Each row gets the full multi-stage evaluation — five llm_judge calls across three stages, plus the two chain evaluations. 28 evaluations total across the suite.

Filter a single row during development

abslang run refund-parametrized.abs.yaml \
  --agent $URL \
  --dataset refund-cases.jsonl \
  --filter "expectedRefundId:R-5512"

This is what you want as QA: the same spec, run against diverse real data, catching regressions scenario by scenario. The session is the what. The dataset is the with what data. The multi-stage evaluation is the how well — at every stage, for every case.

Next: Common mistakes →