awslabs / awslabs/cli-agent-orchestrator
Workflow pre-flight: fail before the run with a named cause (script contract + provider/profile resolution)
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 267
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 70
Description
Parent: #583
## Context
While authoring a real Python workflow (a generic PR-review adjudicate → plan → fix → verify pipeline, exercised live against `awslabs/aidlc-workflows#782`), the majority of failed rounds were not orchestration logic. They were violations of the script↔runtime contract that are statically decidable but only surface as an opaque failure part-way into a live, token-spending run.
Observed, each costing a full run to diagnose:
- **`step_id` charset.** `step_id="draft:reviewer-reply"` returns a bare HTTP 422. The real rule is `WORKFLOW_NAME_RE = ^[A-Za-z0-9_-]{1,64}$`, but the response names no field and no rule.
- **Input type semantics.** A `path`-typed input must be an existing **directory**; a file path has to be declared `"string"`. Discovered by runtime failure, not by `cao workflow validate`.
- **Handle surface.** `StepHandle` exposes `.status`; `.state` raises. No typed stub or documented surface to author against.
- **Profile shadowing + model pins.** A profile in `~/.aws/cli-agent-orchestrator/agent-store/` silently shadows the packaged `agent_store/` profile of the same name. The local `developer.md` pinned `model: openai.gpt-5.6-terra`, valid only under the `codex` provider. Paired with `claude_code` the step returned `API Error (openai.gpt-5.6-terra): 400 Invocation of model ID ... isn't supported` — see the companion issue on in-band errors for why that was *not* reported as a failure.
- **Capability grants.** The packaged `reviewer` profile grants `@builtin, fs_read, fs_list, @cao-mcp-server` — no `execute_bash`, no `fs_write`. A step whose prompt requires a shell command or a file write cannot succeed under it, and nothing says so before the run.
## Gap vs #583
Milestone 1 defines a tested capability contract for **provider-native `/goal` and `/loop` commands**, and the acceptance criteria state that unsupported provider *versions* must fail before execution. That is a different axis from the one above: this is about the **agent profile a step runs under** — which file won the shadowing race, what model it pins, whether that model is servable by the paired provider, and which tools it grants relative to what the step needs.
Nothing in the roadmap resolves or validates that pairing, and Milestone 2's "present the resolved execution plan" cannot present what is never resolved.
## Proposal
Extend `cao workflow validate` and add a run-time pre-flight so both fail **before** any terminal launches, with a named cause:
1. **Script contract, statically.** `step_id` charset/length with the offending value and the rule; duplicate `step_id`s; declared input types against their semantics (`path` = existing directory); unknown `provider` / `agent` names. Ship a typed stub (`.pyi`) or documented surface for `run_step` and `StepHandle` so `.state` is an authoring-time error.
2. **Environment, at resolution time.** For every distinct `(provider, agent)` pair a script can reach: resolve the profile to the file that actually wins, report the shadowed path, and check the profile's model pin against the paired provider's servable models. Fail with e.g. `step 'regen_products': profile 'developer' (~/.aws/cli-agent-orchestrator/agent-store/developer.md, shadows packaged) pins openai.gpt-5.6-terra, which provider 'claude_code' cannot serve`.
3. **Capability requirements, declared.** Let a step declare what it needs (`requires=["execute_bash", "fs_write"]`) and fail pre-flight when the resolved profile does not grant it.
Item 3 is the one that needs a small API addition; items 1 and 2 are checks over information CAO already has.
## Acceptance criteria
- [ ] An invalid `step_id` is rejected by `validate` naming the value and the rule; it never reaches an HTTP 422.
- [ ] A `path` input declared for a file is rejected by `validate` with the correct declaration to use.
- [ ] `validate` resolves every `(provider, agent)` pair a script can reach and reports which profile file won plus any shadowed file.
- [ ] A profile whose model pin the paired provider cannot serve fails pre-flight, before any terminal launches.
- [ ] A step declaring `requires=["execute_bash"]` under a profile that does not grant it fails pre-flight.
- [ ] `StepHandle` and `run_step` have a typed, documented surface; `.state` fails at authoring time.
Contributor guide
Research direction
Start with the `cao workflow validate` entry point and the `run_step`/`StepHandle` surfaces named in the issue, then trace how provider and agent profiles are resolved before a run. Use the listed acceptance criteria as the completion checklist: validation and pre-flight must report named contract, model, shadowing, and capability failures before terminal launches, with typed documentation for the authoring surface.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100