awslabs / awslabs/cli-agent-orchestrator

Re-execute a single workflow step against a recorded run (shorten the authoring loop)

Open
#640 1 comment 0 reactions 1 assignee Claimed by @anilkmr-a2z View on GitHub
enhancement
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

Parent: #583

## Context

Authoring one non-trivial Python workflow (four phases, ~57 steps, three human gates) took **ten live runs**. The orchestration logic was not the cost. The cost was that **the smallest unit of execution is the whole workflow**, so every fix — including one-line prompt fixes — was paid for at full workflow price.

Concrete example. A verification step's prompt had inverted its own subject: it described the uncommitted diff under test as "dirty-worktree drift, not caused by this review", and deferred its verdict to a clean checkout — exonerating the work by construction. The fix was two sentences of prompt. Delivering it required re-reaching step 55 of 57.

The costliest defect class in agent workflows is not the exception — it is **plausible, confidently wrong prose**. It throws nothing, it is only detectable by reading the output, and today each iteration on it costs a full run.

I worked around this by hand-writing a receipt cache (`receipts/.json`, skip-if-present). It made iteration nearly free — and it carried a data-destroying bug, because it keyed on `step_id` alone and had no idea a prompt had changed. That is exactly the mechanism that belongs in the runtime.

## Gap vs #583

The roadmap draws a clean line: **resume** re-drives a frozen run after a transient failure; **repair** changes the artifact and starts a new run. That line is right for operations and wrong for authoring — under it, editing one prompt means a new run that re-pays for all 56 other steps.

Milestone 2 is about the authoring experience and lists "time and manual intervention from request to validated artifact" as a validation signal, but nothing in the roadmap shortens the iteration loop itself.

**This is not the deferred "simulated workflow dry run" non-goal.** Nothing is simulated: it is *real* execution of *one real step* against inputs recorded from a real prior run. It is the authoring counterpart to Milestone 0's journal, and it depends on the same journal.

## Proposal

```
cao workflow step <step_id> --from <run_id> [--prompt-file <edited>]
```

- Resolve the step's recorded inputs from the journal of `<run_id>`, execute that single step live, print its output, and do not mutate the source run.
- Allow the prompt to be overridden so a prompt edit can be evaluated for the price of one step.
- `--write-back` (opt-in) to graft the new result into a *new* run derived from the old journal, so the suffix of the workflow can then be driven from a corrected step.

Adjacent, same journal, higher leverage for long workflows: on a new run, reuse journaled results for steps whose `(prompt, inputs)` fingerprint is unchanged and re-execute from the first changed step onward. Milestone 0 already specifies fingerprint comparison — it currently uses it to *fail loudly*; this asks for the invalidate-and-continue variant, gated behind an explicit flag so operational resume semantics stay strict.

## Acceptance criteria

- [ ] A single step from a completed or failed run can be re-executed live with its recorded inputs.
- [ ] Its prompt can be overridden for that execution without editing the saved artifact.
- [ ] The source run's journal is unmodified by default.
- [ ] Cost of iterating on one prompt is one step, not one workflow.
- [ ] An explicit flag re-runs a workflow reusing journaled results up to the first step whose `(prompt, inputs)` fingerprint changed; default resume semantics are unaffected.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.