aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

fix(agent): a correct no-op is reported as deliverable=lost and classified as a retryable transient fault

Open
#751 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

### Component

Agent (Python runtime)

### Describe the bug

The delivery gate (`_apply_delivery_gate`, `agent/src/pipeline.py:665-728`) cannot distinguish "the agent's work was lost" from "the agent correctly concluded there was nothing to do". Both produce zero commits and no PR, so a correct no-op on a `new_task` is reported to the user as `deliverable=lost` — and the error classifier then tells them it was a **transient infrastructure fault** they should retry.

The docstring already enumerates the sanctioned no-ops it exempts (read-only workflows, artifact workflows, `push_resolve`/`resolve` strategies, `needs_input` clarify-and-hold). A conditional request whose condition is already satisfied — "do X **if** the file lacks Y" where it already has Y — is a fifth sanctioned no-op that is not exempted, and the agent has no way to signal it.

The downstream classification (`cdk/src/handlers/shared/error-classifier.ts:495-512`) then says:

> **The change was not saved** — "…This is usually a transient workspace fault (e.g. the clone ended up in an unexpected directory), not a problem with your request."
> Remedy: "Reply here to try again — a fresh run normally saves the work correctly."

with `retryable: true` and `errorClass: TRANSIENT`. For a correct no-op that advice is actively wrong: retrying can **never** succeed, because the condition will still be satisfied. The user is directed into an unbounded retry loop against a task that already did the right thing.

### Expected behavior

An agent that finished with `agent_status=success`, made no changes, and can articulate *why* no change was needed should terminate as a non-failure outcome (COMPLETED with a "no change required" note, in the spirit of the existing `code_changed=False` "answered" path for pr_iteration), or at minimum as a **non-retryable** failure whose copy says "the agent determined no change was needed" rather than "transient workspace fault, try again".

The `deliverable=lost` copy and `retryable: true` should be reserved for the case it was written for: work that was actually produced and then lost.

### Current behavior

Task `01KZS7NFB480DGF2A6FZ5N3CZD` on `aws-samples/sample-semantic-layer-structured`:

- Request: *"Add a module-level docstring to `agents/shared/advisory.py` if it lacks one. Do not change any logic or tests."*
- `agents/shared/advisory.py` **already has** a module-level docstring, so the correct action was to change nothing — and the agent did exactly that, reporting success after 7 turns.
- Result:

```
Task 01KZS7NFB480DGF2A6FZ5N3CZD — FAILED (7m 21s total)
Last milestone: agent_execution_complete
Reason: agent: The change was not saved — Task did not succeed
(agent_status=success, deliverable=lost): the coding task reported
success but no commit reached the branch and no PR was opened —
the agent's changes did not land in the task's repository.
```

Presented as FAILED / transient / retryable, with $0.19 spent and a remedy that cannot work.

### Reproduction steps

1. Onboard any repo and submit a `new_task` whose instruction is conditional on a state the repo already satisfies, e.g. *"Add a module-level docstring to `` if it lacks one"* against a file that already has one.
2. The agent correctly makes no change and reports success.
3. The task is marked FAILED with `deliverable=lost`, classified TRANSIENT / `retryable: true`, and the user is told to reply to retry.

### Possible solution

Two parts, and the first is the substantive one:

1. **Give a correct no-op a way to be expressed.** The agent needs a terminal outcome for "success, no change required, here is why" that `_apply_delivery_gate` can treat as a sanctioned no-op — analogous to how `needs_input` is exempted today and forced to success right after the gate. Without such a signal the gate cannot tell this case from real lost work, because the observable state (no commit, no PR, agent success) is identical.
2. **Soften the fallback copy.** Until (1) exists, `deliverable=lost` is ambiguous, and the classifier states a specific cause ("the clone ended up in an unexpected directory") with more confidence than the evidence supports. Wording that admits both possibilities — work lost, *or* the agent judged no change necessary — would stop pointing users at a retry that cannot help. Keeping `retryable: true` for the genuinely-lost case is fine; asserting TRANSIENT for both is not.

Found empirically while validating build-regression gating end to end; the task above was a deliberate smoke test, but the misclassification would hit any real conditional request.

Contributor guide

Open the contributing guide

Research direction

Start with _apply_delivery_gate in agent/src/pipeline.py:665-728 and trace how agent_status=success with no commit or PR becomes deliverable=lost. Then inspect cdk/src/handlers/shared/error-classifier.ts:495-512; the work is done when a successful, justified no-op is distinguished from lost work and is reported without retryable transient-fault guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.