awslabs / awslabs/cli-agent-orchestrator
Codex assign can delete active worker when cached status lags visible screen
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 267
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 70
Description
## Summary
Codex workers created through asynchronous `assign()` can receive and start the task, but CAO may still classify the deferred submission as dropped and delete the active worker.
## Environment
- CAO: 2.4.1, main commit `d3efc643b342f262b44eae0c95021539b942d8c1`
- Codex CLI: 0.148.0
- Provider: `codex`
- Backend: tmux
## Reproduction
1. Launch a Codex supervisor through CAO.
2. Have it call `assign()` with a Codex worker and a small task that runs `pwd` and `git branch --show-current`, then callbacks with `send_message`.
3. Observe that the worker reaches a visible `Working (...)` state and can even deliver its callback.
4. The deferred-init confirmation can nevertheless time out against the cached status, log `paste dropped`, retry, and eventually tear down the worker.
Representative ordering from the server log:
```text
Sent input to terminal:
wait_until_status [...]: waiting for {waiting_user_answer, processing, completed}, timeout=8.0s
Delivered 1 message(s) to terminal
Terminal status changed: completed
Terminal status changed: processing
wait_until_status [...]: timeout waiting for {waiting_user_answer, processing, completed}
```
Before the workaround, longer tasks showed `Working (3s)` or `Working (7s)` in the terminal scrollback, but CAO logged three `Deferred assign ... not accepted (paste dropped)` retries and then deleted each worker.
## Cause
`_confirm_worker_started_or_resubmit()` already has a direct visible-screen fallback for an event-driven status-cache race, guarded by `provider.supports_direct_status_probe`. `CodexProvider` supports screen detection and its `get_status()` recognizes the visible `Working (... • esc to interrupt)` indicator, but it does not opt into the direct probe. Therefore the fallback is skipped for Codex.
## Tested fix
```python
class CodexProvider(BaseProvider):
supports_direct_status_probe = True
supports_screen_detection = True
```
After enabling that flag and restarting `cao-server`, the same `assign()` test completed successfully:
- no `paste dropped` retry;
- no worker teardown;
- callback reached the supervisor with the literal command output;
- worker remained available until explicitly deleted.
The cached eight-second wait still emitted a timeout, confirming that the direct probe—not a timing coincidence—prevented the false retry/teardown.
## Suggested regression test
Simulate a Codex deferred assignment where cached status remains `IDLE` while the direct capture reports `PROCESSING`, and assert that confirmation succeeds without re-delivery or deletion.
Contributor guide
Research direction
Start at CodexProvider and _confirm_worker_started_or_resubmit(), then trace the deferred assign confirmation path and the supports_direct_status_probe flag. Reproduce the stale cached IDLE versus direct PROCESSING status described in the issue. Add a regression test that confirms the assignment succeeds without re-delivery or worker deletion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100