code-yeongyu / code-yeongyu/senpi
CI flake: experimental-remote-runtime worker-owned-provider stream fails with socket queue stall (1/3446, rerun green)
- Dominant language
- TypeScript
- Stars
- 429
- Forks
- 98
- Avg merge
- 5h 3m
- Merged PRs (30d)
- 526
Description
## Summary
`test/experimental-remote-runtime.test.ts > experimental durable server composition > streams prompt events through the worker-owned service provider` failed once in CI with an event-sequence assertion while the RPC socket event queue logged stall/overflow warnings. It passed on rerun, on main, and on the branch head — classified as a timing flake, filed for a root-cause fix per the flake policy (fix merged to base, not just reruns).
## Actual
CI run 34799956108, job 103840461978 (PR #1673, merged head 01863bd5e):
- `AssertionError: expected [ 'run_start', 'message_start', …(8) ] to deeply equal ArrayContaining{…}`
- Logged just before: `socket event queue stalled: peer did not drain 57 queued bytes within 50ms` (twice) and `socket event queue overflow: 0 queued + 2166 incoming > 1024`.
## Expected
The worker-owned provider event stream is deterministic under CI load; a slow peer drain does not reorder or drop the expected prompt events.
## Classification evidence (three-step)
1. No intentional change: the PR diff touched only ask-user multi-select keys/state/render/tests + CHANGELOG.
2. BASE contrast: main CI run 34797565211 (post-#1675) all shards green; the PR's pre-merge branch head also had this shard green (10m12s pass).
3. Rerun of the failed job: green. Fail 1 / 3446 passed in the failing attempt.
## Hypothesis
Timing-sensitive RPC socket event-queue liveness guard: the 50ms peer-drain window can abort/close the queue under CI contention, disturbing the event sequence the test asserts. Related families: credit-timeout quarantine, RPC worker stall retries.
## Scope / acceptance
- Reproduce under load or fault injection (slow reader), identify whether the queue-close path can drop/reorder events the test expects.
- Durable fix (backpressure tolerance, drain-window sizing, or test-level event subscription before trigger) merged to main; no test skip, no widened assertion.
Contributor guide
Research direction
Start with test/experimental-remote-runtime.test.ts and reproduce the worker-owned provider stream under load or with a slow reader. Trace the RPC socket event-queue liveness guard, including the 50ms drain window and queue-close path, and compare the related credit-timeout and worker-stall retry behavior. Done means a durable main-branch fix prevents expected prompt events from being dropped or reordered without skipping the test or widening its assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, networking, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100