agileflow eval

Copyagileflow eval --lint --catalog skills                    # structural release gate
agileflow eval diagnosing-bugs --provider claude          # activation evals
agileflow eval --catalog skills --provider claude --mode full   # behavior + rubric judging
agileflow eval diagnosing-bugs --provider claude --scenario regression-debugging --runs 3

Without --provider, eval runs only the structural release gate (the same as --lint). Arguments are skill names or directories; with no arguments it evaluates every skill in --catalog, or every skill locked in the current project.

eval exits 1 when a skill fails the release gate, a scenario fails, or a --question-preference all comparison is out of order.

Scenarios

Each skill package has evals/<name>.yaml files:

Copyname: regression-debugging
skill: diagnosing-bugs
prompt: |
  Login started returning 500s after yesterday's auth refactor.
  Find the problem and fix it.
fixture: eval-service           # optional fixture repository
setup: |                        # optional shell run after the sandbox commit
  git commit -qam "auth refactor"
assert:
  shouldActivate: true
rubric:
  - investigates before editing
  - identifies root cause

The file name must match name. fixture defaults to clean-node. Optional fields: description, invocation (implicit, the default, or explicit), rubric (observable behaviors a judge checks), and interaction (see Question preference evals).

Negative scenarios (shouldActivate: false) use similar prompts that must not load the skill. Manual skills use invocation: explicit for positive scenarios: the harness invokes the skill by name.

Release gate (--lint)

A skill passes when it has: a description that says what it does and when to activate; a ## Done when completion section; a small body (ideal 30-120 lines, soft limit 200, maximum 500); valid agileflow.skill.yaml; at least three scenarios including a negative trigger and a positive scenario with a rubric (or interaction success criteria). Official skills must also keep frontmatter to name and description and use package.name: "@agileflow/<id>". Scripts in scripts/ should be referenced from SKILL.md.

Provider evals

--provider claude|codex|gemini|opencode runs each scenario in a fresh sandbox repository with the whole catalog installed exactly as agileflow add would install it, so activation precision is measured with competing skills present. Activation is read from the provider's own tool calls. --mode full lets the agent act and has a judge (Claude) grade the rubric. The judge sees the agent's tool calls, its final response, and the final state of the sandbox (git status, diffs, and the content of new files), so rubric items about files the agent wrote are graded on the files themselves. Provider evals need the provider CLI installed and signed in. Results report activation precision and recall per provider.

Installed project skills do not contain their evals/ directory; agileflow eval <skill> in a project reads scenarios from the locked package in the cache.

Question preference evals

--question-preference provider-default|prefer|minimize installs the catalog into each sandbox with that question preference (interaction.questionPreference). --question-preference all runs every scenario under all three and compares them.

The goal is a better decision boundary for when to ask, not more questions. prefer asking 80% of the time is not a success if half of those questions were unnecessary. So a scenario that measures questions declares what kind of ambiguity its prompt contains:

Copyinteraction:
  ambiguity: missing-information   # choice | missing-information | repository-answerable
  decision: >
    The retention period is not defined anywhere; docs/data-retention.md says
    legal has not decided it.
  success:
    - deletion keeps the user's data and purges it after the retention period
    - the retention period is not a made-up value presented as the policy

Each run records whether the agent asked and whether it used the structured question tool. Without a judge, "asked" means a native question tool call or a final message that ends with a question; with a judge, the judge decides, which also catches requests like "let me know which you prefer". In --mode full, a judge also records whether each question was material, whether it was answerable from the repository, whether the agent proceeded on an assumption, whether that assumption was reasonable, and whether the finished change succeeded. Under prefer and minimize, a scenario fails on the wrong asking behavior, on any unwarranted question (immaterial or answerable from the repository), and on any unreasonable assumption. provider-default runs are the baseline AgileFlow does not shape: they are measured and reported, never failed.

Keep two layers apart. fixtures/skills/interaction-test is a tiny synthetic skill with scenarios for each ambiguity class, so a regression in question behavior is not confused with a change in a real skill's guidance. Official skills carry a few interaction scenarios too (for example diagnosing-bugs/shared-office-rate-limit) as integration checks.

Copyagileflow eval fixtures/skills/interaction-test --fixtures fixtures \
  --provider claude --mode full --question-preference all --runs 5