awslabs / awslabs/cli-agent-orchestrator

Workflow runtime must classify in-band provider errors as step failures, not completed steps

Open
#638 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.3k
Forks
271
Avg merge
1d 23h
Merged PRs (30d)
70

Description

Parent: #583
Blocks: Milestone 0 (replay-safe Python resume)

Context

Observed while running a real Python workflow: a step failed at the provider level and was reported as an ordinary success.

status: completed
output: "API Error (openai.gpt-5.6-terra): 400 Invocation of model ID openai.gpt-5.6-terra isn't supported."

The transport succeeded — it faithfully delivered a payload — so the run status was completed, with the provider's error text sitting exactly where the model's answer should be. The workflow wrote it into a user-facing deliverable as if it were the answer.

The author-side health check was bool(handle.output) and not state.startswith("error"). Both halves passed. Nothing in the step's own self-report distinguishes "the model answered" from "the model refused to load". The only thing that caught it was an independent check on the step's artifact: the step was supposed to write a file, and no file appeared.

Gap vs #583

Milestone 3 covers retaining error information so a later agent can diagnose a failed run — but this run was never classified as failed. Acceptance criteria cover "unsupported provider versions fail before execution", which is pre-flight for native slash commands, not an in-band model error at invocation time.

This blocks Milestone 0. If a step whose output is a provider error is journaled as completed, then replay-safe resume will faithfully serve that error text on every subsequent run without ever launching a terminal — the fault becomes permanent and free, which is worse than the current behavior. Milestone 0's value depends on "completed" meaning what it says.

Proposal

The runtime, not the workflow author, must classify this. CAO launched the CLI and knows its error signatures.

  1. Add a provider-error classifier per provider adapter — start-anchored signature matching (API Error, 400 Invocation of model ID, provider-specific equivalents), bounded to short outputs so a long legitimate answer that merely quotes an error is not misread as one.
  2. A step so classified is failed, not completed, and handle.ok is False. Expose the classification and the raw output distinctly so an author can still read it.
  3. Never journal such a step. It must retry, not replay its error text.
  4. Surface the distinction in cao workflow status / results so an operator sees failed (provider error) rather than a green step with garbage output.

Acceptance criteria

  • A step whose provider returns an in-band model/API error is reported failed, not completed.
  • handle.ok is False for such a step; the raw provider text remains retrievable.
  • Such a step is not written to the journal and is re-executed on the next run rather than replayed.
  • A legitimate long output that quotes an error string is not misclassified (regression test).
  • status and results distinguish a provider failure from a normal failure and from success.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workflow runtime and provider adapters that classify CLI outputs, then trace handle.ok, journaling, and cao workflow status/results. Add regression coverage for in-band provider errors and long legitimate output that quotes an error, and verify failed steps retain raw text, are not journaled, and are visibly distinguished from success.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.