code-yeongyu / code-yeongyu/senpi
test: sdk-stream-options and interactive-host-runtime give execution-condition-dependent results, making bulk failure counts unusable
- Dominant language
- TypeScript
- Stars
- 429
- Forks
- 98
- Avg merge
- 4h 57m
- Merged PRs (30d)
- 540
Description
## Summary
Two test files in `packages/coding-agent/test/` produce **execution-condition-dependent results**, which makes bulk-run failure counts unusable as a signal. During a regression audit this cost three separate misattributions before it was pinned down, each time sending work at a non-existent bug.
## A. `sdk-stream-options.test.ts` — fails only when run ALONE
The assertion `expect(new Set(attempts).size).toBe(2)` (the 401-retry-uses-a-distinct-credential case) behaves inversely to the usual flake pattern:
| invocation | result |
|---|---|
| file run alone, at unmodified `main` | **1 failed / 13** (measured twice) |
| same file inside a 6-file vitest invocation, same commit | **all 35 passed** |
Measured at `ece35544b` and again in feature worktrees branched from it. So it is not caused by any recent change — running the file in isolation is what surfaces it. That suggests the assertion depends on state or module init established by a sibling test file rather than by its own setup.
## B. `interactive-host-runtime.test.ts` — fails in bulk under ANY competing load
This suite spawns **real RPC hosts** and takes 44-71s.
| condition | result |
|---|---|
| standalone, machine otherwise idle | **27/27**, later **29/29** (measured repeatedly) |
| run alongside one other test file | 10 failed / 27 |
| run while unrelated `gh`/`npm view`/git commands executed concurrently | 18 failed / 29 |
The third row is the important one: merely issuing unrelated shell commands during the run was enough to break it. Sequencing test files is **not** sufficient isolation — the host spawn/readiness path appears to have timing assumptions that competing system load violates.
## Why this matters
Any CI or local run that executes these alongside anything else yields failure counts that are indistinguishable from real regressions. In practice that means:
- reviewers cannot use a bulk run as a gate,
- "pre-existing failure" claims cannot be checked cheaply,
- real regressions can hide inside the noise.
## Suggested direction
- For (A): make the assertion self-sufficient — establish whatever provider/credential state it depends on inside its own setup rather than inheriting it, so isolation and bulk runs agree.
- For (B): replace wall-clock/timing assumptions in host spawn and readiness with explicit readiness signals the test awaits with a generous bound, so the suite is load-tolerant rather than load-sensitive.
Until then, the practical rule is: **verify any suspected failure in these files standalone on an idle machine, and confirm it also occurs at the merge base, before attributing it to a change.**
Found during the second-pass regression audit of `2026.8.29`.
Contributor guide
Research direction
Start with packages/coding-agent/test/sdk-stream-options.test.ts and packages/coding-agent/test/interactive-host-runtime.test.ts, reproducing each failure both alone and under competing load. Trace the credential/provider setup and the real RPC host spawn/readiness path. Done means both suites produce consistent results in isolation and bulk runs, including when the machine is under unrelated load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100