awslabs / awslabs/cli-agent-orchestrator
Workflow runtime must classify in-band provider errors as step failures, not completed steps
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 271
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 70
Description
Parent: #583
Blocks: Milestone 0 (replay-safe Python resume)
Context
Observed while running a real Python workflow: a step failed at the provider level and was reported as an ordinary success.
status: completed
output: "API Error (openai.gpt-5.6-terra): 400 Invocation of model ID openai.gpt-5.6-terra isn't supported."
The transport succeeded — it faithfully delivered a payload — so the run status was completed, with the provider's error text sitting exactly where the model's answer should be. The workflow wrote it into a user-facing deliverable as if it were the answer.
The author-side health check was bool(handle.output) and not state.startswith("error"). Both halves passed. Nothing in the step's own self-report distinguishes "the model answered" from "the model refused to load". The only thing that caught it was an independent check on the step's artifact: the step was supposed to write a file, and no file appeared.
Gap vs #583
Milestone 3 covers retaining error information so a later agent can diagnose a failed run — but this run was never classified as failed. Acceptance criteria cover "unsupported provider versions fail before execution", which is pre-flight for native slash commands, not an in-band model error at invocation time.
This blocks Milestone 0. If a step whose output is a provider error is journaled as completed, then replay-safe resume will faithfully serve that error text on every subsequent run without ever launching a terminal — the fault becomes permanent and free, which is worse than the current behavior. Milestone 0's value depends on "completed" meaning what it says.
Proposal
The runtime, not the workflow author, must classify this. CAO launched the CLI and knows its error signatures.
- Add a provider-error classifier per provider adapter — start-anchored signature matching (
API Error,400 Invocation of model ID, provider-specific equivalents), bounded to short outputs so a long legitimate answer that merely quotes an error is not misread as one. - A step so classified is
failed, notcompleted, andhandle.okisFalse. Expose the classification and the raw output distinctly so an author can still read it. - Never journal such a step. It must retry, not replay its error text.
- Surface the distinction in
cao workflow status/ results so an operator seesfailed (provider error)rather than a green step with garbage output.
Acceptance criteria
- A step whose provider returns an in-band model/API error is reported
failed, notcompleted. -
handle.okisFalsefor such a step; the raw provider text remains retrievable. - Such a step is not written to the journal and is re-executed on the next run rather than replayed.
- A legitimate long output that quotes an error string is not misclassified (regression test).
-
statusand results distinguish a provider failure from a normal failure and from success.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the workflow runtime and provider adapters that classify CLI outputs, then trace handle.ok, journaling, and cao workflow status/results. Add regression coverage for in-band provider errors and long legitimate output that quotes an error, and verify failed steps retain raw text, are not journaled, and are visibly distinguished from success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100