awslabs / awslabs/cli-agent-orchestrator

[Bug] handoff/assign result channel silently truncates or drops a completed worker's reply, then tears down the terminal — long/wide replies force full re-runs (claude_code, tmux)

Open
#728 2 comments 0 reactions 1 assignee Claimed by @sasuw View on GitHub
Dominant language
Python
Stars
1.3k
Forks
271
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Summary

A worker that has finished its task can still fail to hand its result back to the
supervisor. The task itself runs fine; the **return channel** loses it. Two shapes,
seen repeatedly in one session:

1. **Truncation in transit.** The worker's final message arrives cut off
mid-content (observed: cut in the middle of a Markdown validation table). The
supervisor gets a partial message with no verdict.
2. **Empty capture.** The worker completes but no final message is extracted at
all; the supervisor gets nothing.

In both cases the worker terminal is then torn down, so the full text is gone from
the live pane and the only remaining copy is the delete-time snapshot — which is
the same (already truncated) buffer. The supervisor's only recourse is to
re-delegate the entire task. On a review step that means re-running the whole
review.

A closely-related third shape in the same session: a freshly handed-off worker
terminal is **torn down almost immediately, before it produces any output**
("premature handoff teardown"), because the status layer misreads a transient
TUI frame right after the prompt paste as COMPLETED. Evidenced below.

## Impact

- Wasted work: completed reviews and implementations are discarded and re-run
because their result could not be transported.
- Silent: the supervisor cannot distinguish "worker failed" from "worker
succeeded but the channel dropped it", so it re-delegates and can start
duplicate work.
- Recurring, not a one-off — hit it multiple times across consecutive runs.

## Environment

- cli-agent-orchestrator 2.5.0 (installed via `uv tool`)
- Provider: `claude_code`; Claude Code CLI 2.1.259
- Terminal backend: tmux 3.6
- Linux, single host, Python 3.14
- Flow: a supervisor delegating with `handoff` / `assign` to worker profiles;
ad-hoc (not the workflow engine)

## Reproduction

1. Launch a supervisor (`claude_code`) and have it `handoff` a task to a
`claude_code` worker whose reply is **long and contains a wide Markdown
table** (e.g. "review this change and return a verdict plus a table of every
validation command and its result").
2. Let the worker finish normally.
3. Observe the message the supervisor receives: it is either truncated
mid-table or absent, while the worker's own pane (before teardown) showed the
full reply.

Wide/long content and a burst of redraw output at the end of the worker's turn
appear to be the trigger; short replies come back intact.

## What I think is happening (from reading `main` @ c0c9b723222dc85a1ac8f751dc3daf71e5dbcd76)

The worker→supervisor result is produced by **scraping the tmux pane**:

- `CodexProvider` / claude_code provider extract the final message by regex over
the ANSI-stripped rolling buffer — for claude_code, `extract_last_message_from_script`
/ `_extract_last_response_text` anchored on the `⏺` / `●` response marker
(`src/cli_agent_orchestrator/providers/claude_code.py`).
- `run_agent_step` (`src/cli_agent_orchestrator/services/agent_step.py`) extracts
a **transient** `AgentStepResult` and tears the terminal down on success.
- `capture_terminal_snapshot` (`src/cli_agent_orchestrator/services/terminal_service.py`)
writes `.snapshot.json` under `LOG_DIR/terminal/` **before** the
kill — but it is the same rolling buffer, so if the buffer is stale the
snapshot is stale too.

The rolling buffer is fed by `pipe-pane` → FIFO. During this session the server
log repeatedly showed:

```
pipe-pane forwarder for terminal appears stalled (pane advanced, no FIFO data) — re-arming
```

i.e. the FIFO stopped forwarding mid-burst while the pane kept rendering. If the
buffer freezes on mid-burst content and extraction then runs against it, the
result is exactly symptom 1 (truncated at the freeze point) or symptom 2 (marker
never arrived) — and, plausibly, the premature-teardown shape too (status
detection reading a stale frame).

### Evidence — premature teardown (shape 3), captured live

A `claude_code` worker, prompt delivered as one ~7 KB bracketed paste
(`[Pasted text #1 +145 lines]`). Server status-monitor log:

```
07:31:17 send_keys ... keys length: 7148 # the brief, pasted
07:31:21.069 Terminal status changed: processing
07:31:21.345 Terminal status changed: completed # 276 ms later
07:31:21.448 send_keys ... keys length: 5 # CAO sends "/exit"
07:31:24 Stopped pipe-pane / Killed tmux window / Deleted terminal
```

The worker's own captured scrollback for the same window shows it had *just*
started real work when `/exit` arrived:

```
● I'll start by exploring the prepared worktree.
Resume this session with:
claude --resume
```

Total lifetime 13.8 s, no work done, no report. The 276 ms
`processing -> completed` flip is a transient frame (the `❯` input line and the
`● high · /effort` footer are still visible under the paste placeholder) being
read as "agent done", after which the step-teardown path sends `/exit`. This is
the #459 / #91 / #392 family resurfacing on a newer CLI: #459 was last patched
for Claude Code v2.1.212; this is v2.1.259.

A sibling instance in the same session took the other branch: `processing`
flipped back to `idle` in ~140 ms and CAO then logged
`no pickup 8.0s after send (idle, never working)` three times
(`Delivery ... not accepted and provider is not probe-capable; skipping full
re-send to avoid a duplicate task`) — the pasted prompt was never submitted.
That worker sat stuck for ~8 minutes before self-recovering. Same trigger (big
bracketed paste), matches #78.

### Related issues

This overlaps existing issues but I do not think any of them covers this
squarely:

- **#711** — the fifo watchdog single-interval burst→stall→settle blind spot.
This is likely *a* mechanism, but #711 is scoped to the watchdog re-baseline
predicate; it does not address the result channel depending on the buffer
being complete.
- **#388** (closed) — `pipe-pane` silently stops forwarding after an
alternate-screen redraw burst; "historical output missed during the stall is
permanently lost." Same family, different documented trigger, closed.
- **#447 / #715** — durability/async of handoff results and a keyed run record.
Those are about results lost to *MCP timeout* while work continues, and about
retry idempotency — not about a completed reply being scraped incompletely.
- **#570** (closed) — an extraction failure ("no completion marker found")
surfacing as the wrong HTTP status. Confirms "extraction produced no usable
output" is a known, recurring category; here it happens *after* a successful
run and costs the whole result.
- **#78** — long multi-line **inbound** messages not being submitted into an
agent (Enter not registered). Opposite direction, same underlying theme: long
content over the tmux pane I/O path is unreliable. Shape 3's sibling instance
is this bug.
- **#459 / #91 / #392** (all closed) — Claude Code TUI footer / transient state
misdetected as COMPLETED, causing a premature `/exit` that kills the worker.
Shape 3 is this, recurring on Claude Code 2.1.259 (the `● high · /effort`
footer named in #459 is still the frame being misread).
- **#659** — the same premature-teardown effect for the `codex` provider, fixed
by opting the provider into `supports_direct_status_probe`. The `claude_code`
path here logs `provider is not probe-capable` on the re-delivery branch, so
it has no equivalent screen-truth fallback.
- **#287** (closed), **#291**, **#639** — prior "stop scraping the TUI / use a
real result contract" threads; the fix direction below is in that spirit.

## Suggested direction (not prescriptive)

The root problem is that the authoritative result travels only as scraped pane
text. Options, roughly in order of preference:

1. **A result channel that does not depend on the pane.** Let a worker write its
result to a declared path or a structured store, and have `handoff` / `assign`
return *that*, not the scraped buffer. A `pipe-pane` stall then cannot corrupt
or drop it. (Compare #291's git-based result contract and #639's effect
contracts.)
2. **Fail closed on a suspicious extraction.** If the extracted result has no
closing marker, ends mid-line, or ends inside a table/box border, do **not**
tear the terminal down. Keep it and surface an `extraction_incomplete` state
so the supervisor can recover from the still-live pane or re-request — rather
than deleting the only complete copy.
3. **Persist the full ANSI-stripped scrollback at teardown** (not just the
rolling buffer) and make it retrievable by the supervisor by terminal id, so
a dropped result is recoverable without re-running the task.
4. **For shape 3 specifically:** do not act on a `processing -> completed`
transition that lands within a few hundred ms of a prompt paste, and give
the `claude_code` provider the same screen-truth fallback #659 added for
`codex` before sending `/exit`.

## Workaround in use

Until there is a blessed mechanism, the delegation runs entirely through files:

- The supervisor writes each delegation brief to
`$CAO_RUN_DIR//brief-.md` and the `handoff` / `assign` message
is only a short pointer to it. This keeps the pasted prompt small (~120 bytes
instead of ~7 KB), which avoids the redraw burst that triggers shapes 1–3.
- Each worker writes its full report to `$CAO_RUN_DIR//-.md`
and replies with only a verdict-first pointer (`VERDICT: … / report: `);
the supervisor `fs_read`s the file before acting.

Short messages survive the channel; the files survive a stall. This works, but
every operator has to build it by hand in their prompts, and shrinking the
paste only makes shapes 1–3 *less likely*, not impossible.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.