awslabs / awslabs/cli-agent-orchestrator

fifo watchdog: burst→stall→settle within one liveness interval re-baselines onto the post-stall frame and never re-arms

Open
#711 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
271
Avg merge
1d 23h
Merged PRs (30d)
70

Description

### Summary
The fifo pipe-liveness watchdog (`services/fifo_reader.py`, `_check_pipe_liveness`) has one remaining blind spot in the burst→stall→settle family: when the burst, the forwarder stall, and the pane settling into its final frame all happen **inside a single liveness interval** (`PIPE_LIVENESS_CHECK_INTERVAL_S`, default 4s), the next check observes "FIFO delivered bytes since last check" → takes the *healthy: re-baseline* arm → and pins the new baseline to the **post-stall settled frame**. From then on the pane matches the baseline and the FIFO is silent, which is indistinguishable from a legitimately idle terminal, so no strike ever accumulates and the pipe is never re-armed — while the rolling buffer stays permanently stuck on stale mid-burst content and status detection stays wrong for the rest of the terminal's life.

The multi-interval variant of this was fixed by pinning the baseline across checks (the current docstring at `fifo_reader.py:437-452` documents that case well — settle *after* an observed divergence now keeps accumulating strikes). The single-interval variant slips past because the re-baseline decision keys on "any bytes since last check", and the pre-stall burst satisfies it.

### Credit
First observed by @reyph in #463 (the "stall during burst" scenario in their incident notes). Re-filed standalone since the PR has gone quiet.

### Fix sketch
The re-baseline arm needs a stronger predicate than "bytes arrived since last check". Two options that compose: (a) only re-baseline when the pane content at check time is *reachable from* the FIFO-fed buffer tail (cheap suffix check — if the pane shows content the buffer never received, bytes-arrived is not proof of health); (b) re-baseline to the pane frame captured at the moment the last FIFO bytes were seen rather than at check time. Either keeps the idle-terminal false-positive guarantee (both still require FIFO-silent + divergence before re-arming).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read services/fifo_reader.py, especially _check_pipe_liveness and its docstring at lines 437-452, then trace the liveness interval state and existing multi-interval fix. Reproduce the burst→stall→settle sequence within PIPE_LIVENESS_CHECK_INTERVAL_S and verify that the watchdog no longer pins a post-stall baseline while idle terminals remain protected from false re-arms.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.