awslabs / awslabs/cli-agent-orchestrator

[Bug] Duplicate inbox delivery: concurrent deliver_pending picks the same PENDING message twice (pick→mark window unprotected)

Open
#406 2 comments 0 reactions 1 assignee Claimed by @call-me-ram View on GitHub
bug
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Symptom

The same PENDING inbox message is occasionally delivered twice to a terminal, a few milliseconds apart — the worker receives the identical prompt twice (double `send_keys`), which can double-run a task.

## Evidence

Server log `cao_2026-07-09_17-45-39.log` (local run): identical send_keys pairs 4 ms apart at 17:51:09, 17:52:44, and 17:59:45 — three duplicate deliveries to the same terminal in one session.

## Root cause pointer

`src/cli_agent_orchestrator/services/inbox_service.py` — `deliver_pending`'s get-pending → type-into-terminal → mark-DELIVERED sequence is not atomic and has no per-terminal mutual exclusion. Two concurrent `deliver_pending` invocations (e.g. one triggered by a status-monitor idle transition and one by a message-send hook) both read the same PENDING row before either marks it DELIVERED.

## Suggested direction

Either a per-terminal delivery lock (async lock keyed by terminal id held across pick→send→mark), or DB-level claim-before-send (single UPDATE … WHERE status='PENDING' … RETURNING as the pick). We run the per-terminal-lock variant in our fork and it has held under burst tests (message floods while the terminal flaps idle/busy); happy to upstream it as a PR if maintainers want.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.