Inbox messages become permanently undeliverable while the receiving agent is still running

# Inbox messages become permanently undeliverable while the receiving agent is still running

## Summary

A `PENDING` inbox message whose receiver's `terminals` row has been deleted can
never be delivered and never be retried, but it stays in `PENDING` forever with
no log line, no state change, and no way for an operator to tell it apart from a
message that is about to be delivered.

The reconciliation sweep drops these deliberately, and its stated rationale is
sound — you should not keep retrying deliveries to deleted agents. The problem is
that **"the `terminals` row is gone" is not the same as "the agent is gone."**
`cleanup_old_data()` deletes `TerminalModel` rows past `RETENTION_DAYS` without
killing the tmux window or the provider process, so a live, running agent can be
left permanently unreachable while its orchestrator believes the order was sent.

Two separate things are being asked for here: (1) don't let a *live* agent become
unaddressable, and (2) don't let an undeliverable message keep claiming it is
`PENDING`.

## Environment

- `cli-agent-orchestrator` 2.4.1, installed from git `main` @ `e1f64402`
- tmux backend, `claude_code` and `codex` providers
- Linux, single host, ~20 concurrent terminals
- `INBOX_RECONCILE_INTERVAL = 30`, `INBOX_RECONCILE_GRACE_SECONDS = 30`,
`RETENTION_DAYS = 14`

## Mechanism

**1. Delivery is edge-triggered.** `InboxService.deliver_pending()` returns early
unless the receiver is `IDLE`/`COMPLETED` (or eager-eligible). It runs off a
StatusMonitor status transition, and an already-idle terminal emits no new
transition — the case #131 was filed for.

**2. The safety net excludes deleted receivers by design.**
`reconcile_orphaned_messages()` (`services/inbox_service.py:154`) iterates
`list_pending_receiver_ids_older_than()` (`clients/database.py:1328`):

```python
db.query(InboxModel.receiver_id)
.join(TerminalModel, TerminalModel.id == InboxModel.receiver_id) # inner join
.filter(
InboxModel.status == MessageStatus.PENDING.value,
InboxModel.created_at < cutoff,
)
```

Its docstring states the intent plainly:

> The join on `terminals` drops messages whose receiver terminal no longer
> exists, so the sweep does not keep retrying deliveries to deleted agents.

**3. But a row can be deleted while the process keeps running.**
`cleanup_service.cleanup_old_data()` runs
`db.query(TerminalModel).filter(TerminalModel.last_active < cutoff).delete()`.
It stops the FIFO reader and clears the status monitor, but it never calls
`terminal_service.delete_terminal()` — so the tmux window and the provider
process survive the removal of their registry row. After that point the agent is
alive, reachable by a human attaching to the pane, and unreachable by the
orchestrator forever.

Steps 2 and 3 compose into a silent, permanent drop.

## Evidence

On this host, **22 of 22** messages sitting in `pending`/`failed` had already lost
their receiver's `terminals` row — i.e. every stuck message had crossed the line
into permanent undeliverability. None had been logged as dropped. Ages ranged
from 20 hours to 12 days.

Two concrete examples, both `failed`, both from a still-running top-level
supervisor that has no way to know they never arrived:

| sent | content | outcome |
|---|---|---|
| `2026-08-27 22:10:30` | `SCOPE AMENDMENT … supersedes part of section 4, section 7 and section 8 of your brief` — conditionally lifting a read-only constraint on one file | worker delivered its callback at `22:22:11`, having independently concluded the file did not need to change |
| `2026-08-28 06:10:17` | `STEER — tooling only … STOP searching for or installing parsers, browsers` — a stop-loss redirect | worker delivered its callback at `06:12:00`, having independently done what the steer asked |

**In both of these cases the loss happened to be harmless**, and I want to be
accurate about that rather than overstate the impact: the first order was
conditional and its condition was not met, and the second arrived under two
minutes before the worker finished on its own. But neither outcome was by
design — an unconditional order, or the same order sent ten minutes earlier,
would have been lost just as silently.

The operator-visible symptom this produces is a review round that appears to
stall for no reason: the supervisor believes it dispatched a correction, the
worker never saw one, and nothing anywhere reports a delivery failure.

## Secondary observation

Messages set to `FAILED` in the generic `except` branch of `deliver_pending()`
(`inbox_service.py:139`) are never retried by any path, and the exception detail
goes only to the log line, not to the row. A `failed` row records that something
went wrong but not what, and nothing surfaces it.

## Suggested directions

Not prescriptive — any one of these would close the silent-loss part:

1. **Make retention consistent with liveness.** Have `cleanup_old_data()` call
`terminal_service.delete_terminal()` rather than deleting the row directly, so
a reaped registry row implies a reaped process. This also fixes the orphaned
tmux windows the current path leaves behind, and would make the inner join's
assumption true.
2. **Give the drop a name.** Add a terminal `UNDELIVERABLE` (or `EXPIRED`) message
state and transition rows into it when their receiver is gone, so `PENDING`
stops meaning two different things. A single `LEFT JOIN … WHERE TerminalModel.id
IS NULL` sweep would find them.
3. **Log it once.** Even without a schema change, emitting one warning when a
pending message's receiver has disappeared would make this diagnosable instead
of invisible.
4. **Record the failure reason** on `FAILED` rows so an operator can tell a
transient pane error from a real one.

Happy to test a patch on this host — it reproduces continuously here.