awslabs / awslabs/cli-agent-orchestrator

[Bug] cao session send returns the previous turn's answer or a mid-flight frame (#407 recurring, on both status paths)

Open
#735 0 comments 0 reactions 1 assignee Claimed by @guojing1217 View on GitHub
bug
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Summary

This is #407's symptom again, on a different caller. `cao session send` posts the input,
sleeps 3 seconds, then polls a status that describes the terminal's *current frame* rather than
the turn just sent. If a completion marker is on that frame — typically the one the previous
turn left — the command returns immediately and prints whatever the buffer holds: the previous
turn's answer, or a partially rendered frame of the turn still running.

The agent is healthy in every case below. Only the CLI's read of it is wrong.

#407 ("stale cached COMPLETED satisfies the next wait_until_status immediately (returns
previous turn's output)") was closed as completed on 2026-07-20 by PR #480, which added a
content-based staleness guard to `claude_code`. **I measured it in both status modes and it
reproduces in both**, so whatever #480 closed, it does not cover this path.

## Version

`2.5.0`. Measured on a working tree two files ahead of `main` (`3eafb6d6`) in `src/`, neither
on the path under test: `utils/terminal.py` adds an optional `read_status=None` parameter to
`poll_until_done` (and `cao session send` passes no such argument, so it runs `main`'s body
unchanged), and `constants.py` adds three unrelated env-var names. `cli/commands/session.py`,
`services/status_monitor.py`, `services/terminal_service.py`, `providers/claude_code.py`,
`providers/base.py` and `api/main.py` are byte-identical to `main`. Provider `claude_code`, tmux backend, macOS 15 (Darwin
25.6.0), Python 3.12. Tested with `CAO_PYTE_STATUS` at its default (`true`) and again with
`false`.

## Reproduction

No cluster, no remote anything — plain local CAO.

```bash
cao-server &
cao launch --agents developer --provider claude_code --headless --auto-approve \
--working-directory /tmp/cao-repro
cao session send "Reply with exactly this one word and nothing else: WARM"
# then, as a trial, one turn with ~20s of real work in it:
cao session send "Run the shell command 'sleep 20; echo ON2' and then reply with exactly the word it printed."
```

A trial passes only if the send takes at least 20 seconds and prints the token. **Wait for the
agent to actually finish between trials** — see the pile-up note below, or the trials stop
being independent.

### Measured, three settled trials per mode

| mode | trial | send returned | printed | verdict |
|---|---|---|---|---|
| `CAO_PYTE_STATUS=true` (default) | 1 | 31s | `ON1` | correct |
| | 2 | **7s** | tool line + `✶ Combobulating… (3s · ↓ 21 tokens)` | wrong — mid-flight frame |
| | 3 | **7s** | `ON2` | wrong — previous turn's answer |
| `CAO_PYTE_STATUS=false` | 1 | 31s | `OFF1` | correct |
| | 2 | **8s** | tool line + `✽ Embellishing… (3s · ↓ 12 tokens)` | wrong — mid-flight frame |
| | 3 | 29s | `OFF3` | correct |

Both symptoms, both modes, intermittent in both. In every failing trial the correct answer
appeared in the pane about 24 seconds after the command had already returned, so the turn
itself was fine.

An earlier five-turn run showed the same on a longer horizon: five sends in a row each
returned in 7–8 seconds and each printed `WARM`, the answer from the warm-up turn.

### The failure compounds when scripted

Because the send returns while the agent is still working, the next send in a loop pastes a
new prompt into a busy agent. The agent batches them. In that five-turn run the pane ended up
with five prompts typed at `❯` and exactly one `⏺ WARM` response — four tasks were never
answered, and nothing reported an error. This is the shape that loses work in a fan-out.

## Cause

[`cli/commands/session.py:246-251`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/cli/commands/session.py#L246-L251)
sleeps a flat 3 seconds after posting the input, then calls `poll_until_done`, which returns on
the first COMPLETED reading — deliberately, because "a single reading is trustworthy"
([`utils/terminal.py:253`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/utils/terminal.py#L253)).
The status it reads is derived from the terminal's frame, so COMPLETED means "a completion
marker is on screen", not "the turn I sent has finished". The output read has the same gap:
`GET /terminals/{id}/output?mode=last` returns the newest response block it can find, with no
notion of "since my send".

Nothing correlates either read with the input just posted. The 3-second sleep is the only thing
between them, which is why a turn finishing inside 3 seconds usually looks right and a slower
one does not.

Two observations from the source, offered as leads rather than as a diagnosis — I could not
confirm either is *the* cause, since the defect survives with pyte off:

- **The defence written for exactly this is switched off everywhere.**
[`providers/base.py:254`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/providers/base.py#L254)
defines `assume_processing_on_dispatch`, documented as: *"Full-screen TUIs that can remain
visually unchanged just after submission opt in so callers cannot observe the previous turn's
cached COMPLETED state as the new turn's result."* It returns `False`, and **no provider
overrides it** — grep finds only the base definition. `terminal_service.send_input` gates
`notify_input_sent(assume_processing=True)` on that property, so the branch never runs.
Otherwise `notify_input_sent` only arms a revert, and `clear_rolling_buffer`'s docstring
states it preserves `_last_status`.
- **#480's guard is on one of the two detectors.** The tail-hash check
(`_input_generation > 0 and _snapshot_tail_hash is not None → PROCESSING`) lives in
`claude_code.get_status()`, the raw-buffer path. With pyte on, StatusMonitor routes to
`get_status_from_screen()` instead, whose 91 lines contain no `_snapshot_tail_hash`,
`_input_generation` or `_tail_hash`. That asymmetry looked like the answer until the pyte-off
arm failed too — the guard only holds while the tail is *unchanged*, and it releases as soon
as the agent renders its first tool line, after which frame shape decides again.

## Suggested direction

Make "done" refer to the send, rather than adding another frame heuristic:

1. Have `POST /terminals/{id}/input` return a monotonically increasing turn or response
sequence number, and let `poll_until_done` and `?mode=last` both take it: wait for a
response *after* that point, and return that response.
2. Failing a server-side sequence, have the CLI record what `?mode=last` returned immediately
before the send and treat an identical value as "not yet" rather than as the answer.
3. Require the status to leave `{idle, completed}` at least once before a COMPLETED reading is
honoured — the same "has started" gate the IDLE path already has. Cheapest, and still loses
to a turn that renders no processing frame within the poll interval.
4. Independently of the above: either set `assume_processing_on_dispatch` on the TUI providers
or delete it, since as shipped no provider can reach it.

## Affected callers

- `cao session send` — everything above.
- `cao launch` at [`launch.py:376`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/cli/commands/launch.py#L376)
shares `poll_until_done`, though its first turn has no previous marker to inherit.

## Related

- **#407** (closed, completed 2026-07-20) — same symptom, reported against
`terminal_service.send_input` + `wait_until_status`. Fixed by **#480**. This is that failure
on the CLI's own poller, and it survives in both status modes.
- **#728** (opened 2026-09-03) — its shape 3 is the same gap with teardown attached: a
`processing → completed` flip 276 ms after a paste is read as done and the worker is killed.
This report is the quieter form: no handoff, no assign, nothing torn down, just a wrong
answer printed. The trigger differs too — a marker left by an *earlier* turn rather than a
transient frame in the current one — so a fix scoped to "ignore a completion within N ms of a
paste" would not catch it.
- **#537** (closed) — the mirror image on `codex`: a sync send that sticks at PROCESSING
instead of returning early.
- **#659** — the same premature-completion effect on `codex`, fixed by opting the provider into
`supports_direct_status_probe`. `claude_code` does not set that flag, but it would not help
here: that gap is cached status disagreeing with the screen, and here they agree and both
describe the old turn.
- **#459 / #91 / #392** (closed) — individual frames that misreported COMPLETED, each fixed by
tightening what counts as a marker. This is the case tightening cannot reach: the marker is
real, it is just old.
- **#287** (closed, completed) — pyte rendering, on by default in the runs above.

The pattern across those is one fix per frame. What is still missing is a way to ask "did *my* send finish", which is why suggestion 1 is the one I would rather build.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.