awslabs / awslabs/cli-agent-orchestrator
[Bug] Duplicate inbox delivery: concurrent deliver_pending picks the same PENDING message twice (pick→mark window unprotected)
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 267
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 70
Description
## Symptom
The same PENDING inbox message is occasionally delivered twice to a terminal, a few milliseconds apart — the worker receives the identical prompt twice (double `send_keys`), which can double-run a task.
## Evidence
Server log `cao_2026-07-09_17-45-39.log` (local run): identical send_keys pairs 4 ms apart at 17:51:09, 17:52:44, and 17:59:45 — three duplicate deliveries to the same terminal in one session.
## Root cause pointer
`src/cli_agent_orchestrator/services/inbox_service.py` — `deliver_pending`'s get-pending → type-into-terminal → mark-DELIVERED sequence is not atomic and has no per-terminal mutual exclusion. Two concurrent `deliver_pending` invocations (e.g. one triggered by a status-monitor idle transition and one by a message-send hook) both read the same PENDING row before either marks it DELIVERED.
## Suggested direction
Either a per-terminal delivery lock (async lock keyed by terminal id held across pick→send→mark), or DB-level claim-before-send (single UPDATE … WHERE status='PENDING' … RETURNING as the pick). We run the per-terminal-lock variant in our fork and it has held under burst tests (message floods while the terminal flaps idle/busy); happy to upstream it as a PR if maintainers want.
Contributor guide
Assessment
This issue has not been assessed yet.