code-yeongyu / code-yeongyu/senpi

CI flake: experimental-remote-runtime worker-owned-provider stream fails with socket queue stall (1/3446, rerun green)

Open
#1,676 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
429
Forks
98
Avg merge
5h 3m
Merged PRs (30d)
526

Description

## Summary

`test/experimental-remote-runtime.test.ts > experimental durable server composition > streams prompt events through the worker-owned service provider` failed once in CI with an event-sequence assertion while the RPC socket event queue logged stall/overflow warnings. It passed on rerun, on main, and on the branch head — classified as a timing flake, filed for a root-cause fix per the flake policy (fix merged to base, not just reruns).

## Actual

CI run 34799956108, job 103840461978 (PR #1673, merged head 01863bd5e):
- `AssertionError: expected [ 'run_start', 'message_start', …(8) ] to deeply equal ArrayContaining{…}`
- Logged just before: `socket event queue stalled: peer did not drain 57 queued bytes within 50ms` (twice) and `socket event queue overflow: 0 queued + 2166 incoming > 1024`.

## Expected

The worker-owned provider event stream is deterministic under CI load; a slow peer drain does not reorder or drop the expected prompt events.

## Classification evidence (three-step)

1. No intentional change: the PR diff touched only ask-user multi-select keys/state/render/tests + CHANGELOG.
2. BASE contrast: main CI run 34797565211 (post-#1675) all shards green; the PR's pre-merge branch head also had this shard green (10m12s pass).
3. Rerun of the failed job: green. Fail 1 / 3446 passed in the failing attempt.

## Hypothesis

Timing-sensitive RPC socket event-queue liveness guard: the 50ms peer-drain window can abort/close the queue under CI contention, disturbing the event sequence the test asserts. Related families: credit-timeout quarantine, RPC worker stall retries.

## Scope / acceptance

- Reproduce under load or fault injection (slow reader), identify whether the queue-close path can drop/reorder events the test expects.
- Durable fix (backpressure tolerance, drain-window sizing, or test-level event subscription before trigger) merged to main; no test skip, no widened assertion.

Contributor guide

Open the contributing guide

Research direction

Start with test/experimental-remote-runtime.test.ts and reproduce the worker-owned provider stream under load or with a slow reader. Trace the RPC socket event-queue liveness guard, including the 50ms drain window and queue-close path, and compare the related credit-timeout and worker-stall retry behavior. Done means a durable main-branch fix prevents expected prompt events from being dropped or reordered without skipping the test or widening its assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, networking, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.