awslabs / awslabs/cli-agent-orchestrator
fifo watchdog: burst→stall→settle within one liveness interval re-baselines onto the post-stall frame and never re-arms
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 271
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 70
Description
### Summary
The fifo pipe-liveness watchdog (`services/fifo_reader.py`, `_check_pipe_liveness`) has one remaining blind spot in the burst→stall→settle family: when the burst, the forwarder stall, and the pane settling into its final frame all happen **inside a single liveness interval** (`PIPE_LIVENESS_CHECK_INTERVAL_S`, default 4s), the next check observes "FIFO delivered bytes since last check" → takes the *healthy: re-baseline* arm → and pins the new baseline to the **post-stall settled frame**. From then on the pane matches the baseline and the FIFO is silent, which is indistinguishable from a legitimately idle terminal, so no strike ever accumulates and the pipe is never re-armed — while the rolling buffer stays permanently stuck on stale mid-burst content and status detection stays wrong for the rest of the terminal's life.
The multi-interval variant of this was fixed by pinning the baseline across checks (the current docstring at `fifo_reader.py:437-452` documents that case well — settle *after* an observed divergence now keeps accumulating strikes). The single-interval variant slips past because the re-baseline decision keys on "any bytes since last check", and the pre-stall burst satisfies it.
### Credit
First observed by @reyph in #463 (the "stall during burst" scenario in their incident notes). Re-filed standalone since the PR has gone quiet.
### Fix sketch
The re-baseline arm needs a stronger predicate than "bytes arrived since last check". Two options that compose: (a) only re-baseline when the pane content at check time is *reachable from* the FIFO-fed buffer tail (cheap suffix check — if the pane shows content the buffer never received, bytes-arrived is not proof of health); (b) re-baseline to the pane frame captured at the moment the last FIFO bytes were seen rather than at check time. Either keeps the idle-terminal false-positive guarantee (both still require FIFO-silent + divergence before re-arming).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read services/fifo_reader.py, especially _check_pipe_liveness and its docstring at lines 437-452, then trace the liveness interval state and existing multi-interval fix. Reproduce the burst→stall→settle sequence within PIPE_LIVENESS_CHECK_INTERVAL_S and verify that the watchdog no longer pins a post-stall baseline while idle terminals remain protected from false re-arms.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100