awslabs / awslabs/cli-agent-orchestrator
[Feat] Terminal durability: re-adopt live terminals after cao-server restart + early snapshots for crash paths
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 267
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 70
Description
## Problem
Two gaps make orchestras fragile across cao-server restarts/upgrades and host crashes:
1. `fifo_manager.create_reader()` has exactly one caller (create_terminal), so after a server restart live tmux agents become half-adopted zombies: the pane keeps running but `.log` stops growing and status detection observes nothing. The lifespan never rescans the `terminals` table.
2. `.scrollback`/`.snapshot.json` are only written on CLEAN deletion — a crash, `tmux kill-server`, or reboot leaves no restore metadata at all (docs/terminal-lifecycle.md's table is also out of date on the shutdown rows).
## Proposal
- Startup re-adoption in the lifespan (after the event-bus consumers): rows whose tmux window survived get their FIFO reader recreated and pipe-pane restarted (stop-then-start; plus a repaint nudge so the fresh rolling buffer isn't empty); rows whose window is gone are finalized (scrollback recovered from the ANSI-stripped log, row dropped).
- Write the snapshot (static launch metadata) at terminal CREATION and refresh it on clean delete.
Implementation ready in a fork with tests; PR follows. Verified live: killed cao-server mid-handoff, restarted, terminals re-adopted and the pending handoff completed.
Contributor guide
Research direction
Start with fifo_manager.create_reader() and the lifespan startup sequence after the event-bus consumers; inspect how terminal rows and tmux windows are currently handled. Review docs/terminal-lifecycle.md and the existing tests, then verify that surviving terminals resume logging and status detection, missing windows are finalized, and snapshots exist from creation through clean deletion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, devtools
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100