awslabs / awslabs/cli-agent-orchestrator

Codex assign can delete active worker when cached status lags visible screen

Open
#659 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Summary

Codex workers created through asynchronous `assign()` can receive and start the task, but CAO may still classify the deferred submission as dropped and delete the active worker.

## Environment

- CAO: 2.4.1, main commit `d3efc643b342f262b44eae0c95021539b942d8c1`
- Codex CLI: 0.148.0
- Provider: `codex`
- Backend: tmux

## Reproduction

1. Launch a Codex supervisor through CAO.
2. Have it call `assign()` with a Codex worker and a small task that runs `pwd` and `git branch --show-current`, then callbacks with `send_message`.
3. Observe that the worker reaches a visible `Working (...)` state and can even deliver its callback.
4. The deferred-init confirmation can nevertheless time out against the cached status, log `paste dropped`, retry, and eventually tear down the worker.

Representative ordering from the server log:

```text
Sent input to terminal:
wait_until_status [...]: waiting for {waiting_user_answer, processing, completed}, timeout=8.0s
Delivered 1 message(s) to terminal
Terminal status changed: completed
Terminal status changed: processing
wait_until_status [...]: timeout waiting for {waiting_user_answer, processing, completed}
```

Before the workaround, longer tasks showed `Working (3s)` or `Working (7s)` in the terminal scrollback, but CAO logged three `Deferred assign ... not accepted (paste dropped)` retries and then deleted each worker.

## Cause

`_confirm_worker_started_or_resubmit()` already has a direct visible-screen fallback for an event-driven status-cache race, guarded by `provider.supports_direct_status_probe`. `CodexProvider` supports screen detection and its `get_status()` recognizes the visible `Working (... • esc to interrupt)` indicator, but it does not opt into the direct probe. Therefore the fallback is skipped for Codex.

## Tested fix

```python
class CodexProvider(BaseProvider):
supports_direct_status_probe = True
supports_screen_detection = True
```

After enabling that flag and restarting `cao-server`, the same `assign()` test completed successfully:

- no `paste dropped` retry;
- no worker teardown;
- callback reached the supervisor with the literal command output;
- worker remained available until explicitly deleted.

The cached eight-second wait still emitted a timeout, confirming that the direct probe—not a timing coincidence—prevented the false retry/teardown.

## Suggested regression test

Simulate a Codex deferred assignment where cached status remains `IDLE` while the direct capture reports `PROCESSING`, and assert that confirmation succeeds without re-delivery or deletion.

Contributor guide

Open the contributing guide

Research direction

Start at CodexProvider and _confirm_worker_started_or_resubmit(), then trace the deferred assign confirmation path and the supports_direct_status_probe flag. Reproduce the stale cached IDLE versus direct PROCESSING status described in the issue. Add a regression test that confirms the assignment succeeds without re-delivery or worker deletion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.