aws / aws/tools-for-devops-agent
PFR: expose pluggable agent runners via an `--agent` flag (so `functional`/`trigger`/`report` can use non-Claude CLIs)
- Dominant language
- Python
- Stars
- 63
- Forks
- 51
- Avg merge
- 5d 20h
- Merged PRs (30d)
- 10
Description
### Summary
The `functional`, `trigger`, `report`, and `compare` commands currently drive the
`claude` CLI exclusively. The codebase already contains the abstraction needed to
support other agent CLIs — `AgentRunner` (ABC), a `register_runner()` / `get_runner()`
registry, and per-runner skill-injection hooks — but only `ClaudeRunner` is registered,
and there is no user-facing way to select a different runner. As a result, anyone
without the Anthropic `claude` CLI installed cannot run the agent-backed evaluations
(functional / trigger / report), even though `audit` / `init` / `snapshot` / `regression`
work fine without an agent.
This request is to **finish wiring up the abstraction that already exists**: add a way
to select a registered runner (e.g. an `--agent NAME` flag), and document the
`AgentRunner` contract so third parties can register their own runners.
### Why this matters
- The agent-backed commands are gated on one specific vendor CLI (`claude`). Teams
standardized on other coding agents (Amazon Q / Kiro CLI, aider, Codex, Cursor CLI,
etc.) currently can't produce a functional/trigger/unified grade at all.
- The design clearly anticipated this — `agent_runner.py` documents the interface as
"enables support for any agent CLI by implementing the AgentRunner interface and
registering it with register_runner()". Today that extensibility isn't reachable from
the CLI; a user would have to monkey-patch the registry in a wrapper script.
- CI adoption: the GitHub Actions reusable workflow (`skill-eval.yml`) is similarly
Claude-bound. A runner selector would let orgs run the same gate with whatever agent
they already have credentials for.
### Current behavior (as of `main`)
- `skill_eval/agent_runner.py` defines:
- `AgentRunner` (ABC) with `check_available()`, `run_prompt()`, `parse_output()`,
and a default `total_tokens()`.
- `register_runner(name, cls)` and `get_runner(name="claude")` — a working registry.
- `ClaudeRunner`, registered as `"claude"`, which shells out to `claude -p ...`
with `--append-system-prompt ` for the with-skill arm and
`--output-format stream-json` parsing.
- `get_runner()` defaults to `"claude"` and nothing passes a different name, so the
functional/trigger/report paths always resolve to `ClaudeRunner`.
- There is no `--agent` (or equivalent) CLI option, and no environment variable, to
select a different registered runner.
### Proposed behavior
1. **Runner selector on the agent-backed commands.** Add `--agent NAME` (default
`claude`) to `functional`, `trigger`, `report`, and `compare`, threading `NAME`
through to `get_runner(NAME)`. Optionally honor a `SKILL_EVAL_AGENT` env var so CI
can set it once.
2. **Clear error when a runner isn't registered/available.** If `--agent foo` is
unknown, list the registered runners (the registry already produces this message);
if the selected runner's `check_available()` fails, surface its install hint (as
`ClaudeRunner` already does).
3. **Document the `AgentRunner` contract** in the README/AGENTS.md: the four methods to
implement, the expected `parse_output()` shape (`events`, `tool_calls`, `text`,
`token_counts`), the skill-injection convention, and a minimal example of a custom
runner registered via `register_runner()`. Include what CLI shape a runner is
expected to wrap (non-interactive "prompt in → machine-readable transcript out",
ideally a streaming-JSON mode, plus a way to inject skill/system context).
4. **(Optional) Entry-point discovery** so runners can be contributed as separate
packages without editing this repo (e.g. a `skill_eval.runners` entry-point group).
### Motivating example
We built and validated a skill against a real agent and want to produce the unified
grade the project gates on, but the `claude` CLI isn't available in our environment; we
do have another agent CLI. With `--agent ` plus a small `AgentRunner` subclass, we
could run `skill-eval report ./skills/ --agent ` and get audit + functional
+ trigger in one grade. (Happy to contribute a reference runner as a follow-up PR — see
below.)
### Backwards compatibility
Fully backwards compatible: `--agent` defaults to `claude`, so existing invocations and
the CI workflow behave exactly as today.
### Willing to contribute
Yes — if the maintainers are open to it, we're happy to open a PR that (a) adds the
`--agent` flag threaded through the four commands, (b) adds the unknown-runner and
not-available error handling, and (c) documents the `AgentRunner` contract with a small
example runner. We'd like to align on the flag name and the runner-discovery approach
(explicit `register_runner()` vs. entry points) in this issue before sending code, per
CONTRIBUTING.md.
### Environment
- `sample-agent-skill-eval` @ `main`
- Observed in `skill_eval/agent_runner.py`, `skill_eval/_claude.py`, `skill_eval/cli.py`
Contributor guide
Research direction
Start with skill_eval/agent_runner.py to understand AgentRunner, the runner registry, and ClaudeRunner, then trace agent selection through skill_eval/cli.py and skill_eval/_claude.py. Review README/AGENTS.md and CONTRIBUTING.md before choosing the flag and discovery approach. Done means the four commands can select registered runners, unavailable or unknown runners report useful errors, and the AgentRunner contract is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, developer-experience, documentation, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100