aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

fix(agent): a correct no-op is reported as deliverable=lost and classified as a retryable transient fault

Đang mở
#751 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
TypeScript
Star
143
Fork
46
Merge trung bình
3 ngày 10 giờ
Pull request đã merge (30 ngày)
24

Mô tả

### Component

Agent (Python runtime)

### Describe the bug

The delivery gate (`_apply_delivery_gate`, `agent/src/pipeline.py:665-728`) cannot distinguish "the agent's work was lost" from "the agent correctly concluded there was nothing to do". Both produce zero commits and no PR, so a correct no-op on a `new_task` is reported to the user as `deliverable=lost` — and the error classifier then tells them it was a **transient infrastructure fault** they should retry.

The docstring already enumerates the sanctioned no-ops it exempts (read-only workflows, artifact workflows, `push_resolve`/`resolve` strategies, `needs_input` clarify-and-hold). A conditional request whose condition is already satisfied — "do X **if** the file lacks Y" where it already has Y — is a fifth sanctioned no-op that is not exempted, and the agent has no way to signal it.

The downstream classification (`cdk/src/handlers/shared/error-classifier.ts:495-512`) then says:

> **The change was not saved** — "…This is usually a transient workspace fault (e.g. the clone ended up in an unexpected directory), not a problem with your request."
> Remedy: "Reply here to try again — a fresh run normally saves the work correctly."

with `retryable: true` and `errorClass: TRANSIENT`. For a correct no-op that advice is actively wrong: retrying can **never** succeed, because the condition will still be satisfied. The user is directed into an unbounded retry loop against a task that already did the right thing.

### Expected behavior

An agent that finished with `agent_status=success`, made no changes, and can articulate *why* no change was needed should terminate as a non-failure outcome (COMPLETED with a "no change required" note, in the spirit of the existing `code_changed=False` "answered" path for pr_iteration), or at minimum as a **non-retryable** failure whose copy says "the agent determined no change was needed" rather than "transient workspace fault, try again".

The `deliverable=lost` copy and `retryable: true` should be reserved for the case it was written for: work that was actually produced and then lost.

### Current behavior

Task `01KZS7NFB480DGF2A6FZ5N3CZD` on `aws-samples/sample-semantic-layer-structured`:

- Request: *"Add a module-level docstring to `agents/shared/advisory.py` if it lacks one. Do not change any logic or tests."*
- `agents/shared/advisory.py` **already has** a module-level docstring, so the correct action was to change nothing — and the agent did exactly that, reporting success after 7 turns.
- Result:

```
Task 01KZS7NFB480DGF2A6FZ5N3CZD — FAILED (7m 21s total)
Last milestone: agent_execution_complete
Reason: agent: The change was not saved — Task did not succeed
(agent_status=success, deliverable=lost): the coding task reported
success but no commit reached the branch and no PR was opened —
the agent's changes did not land in the task's repository.
```

Presented as FAILED / transient / retryable, with $0.19 spent and a remedy that cannot work.

### Reproduction steps

1. Onboard any repo and submit a `new_task` whose instruction is conditional on a state the repo already satisfies, e.g. *"Add a module-level docstring to `` if it lacks one"* against a file that already has one.
2. The agent correctly makes no change and reports success.
3. The task is marked FAILED with `deliverable=lost`, classified TRANSIENT / `retryable: true`, and the user is told to reply to retry.

### Possible solution

Two parts, and the first is the substantive one:

1. **Give a correct no-op a way to be expressed.** The agent needs a terminal outcome for "success, no change required, here is why" that `_apply_delivery_gate` can treat as a sanctioned no-op — analogous to how `needs_input` is exempted today and forced to success right after the gate. Without such a signal the gate cannot tell this case from real lost work, because the observable state (no commit, no PR, agent success) is identical.
2. **Soften the fallback copy.** Until (1) exists, `deliverable=lost` is ambiguous, and the classifier states a specific cause ("the clone ended up in an unexpected directory") with more confidence than the evidence supports. Wording that admits both possibilities — work lost, *or* the agent judged no change necessary — would stop pointing users at a retry that cannot help. Keeping `retryable: true` for the genuinely-lost case is fine; asserting TRANSIENT for both is not.

Found empirically while validating build-regression gating end to end; the task above was a deliberate smoke test, but the misclassification would hit any real conditional request.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu với _apply_delivery_gate trong agent/src/pipeline.py:665-728 và theo dõi cách agent_status=success không có commit hoặc PR trở thành deliverable=lost. Sau đó kiểm tra cdk/src/handlers/shared/error-classifier.ts:495-512; công việc hoàn tất khi một no-op thành công và có lý do chính đáng được phân biệt với công việc bị mất và được báo cáo mà không kèm hướng dẫn về lỗi tạm thời có thể thử lại.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python, typescript
Lĩnh vực
ai, backend
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
52/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.