test(cli): suite is POSIX-only — gh stubs never engage on Windows, so eight tests call the real GitHub API
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 71
- Forks
- 64
- Avg merge
- 15h 38m
- Merged PRs (30d)
- 66
Description
Summary
The CLI test suite is POSIX-only in three independent ways, and the sharpest consequence is not "tests fail on Windows" — it is that eight tests silently bypass their own gh stub and call the real GitHub API from the developer's machine.
Run context: Windows 11, Node 22.23.2, pnpm 11.20.0 (stable, no flags), clean main, pnpm install --frozen-lockfile green. node --test test/*.test.mjs in packages/cli: 86 pass / 11 fail.
Failure class 1 — the gh stub never engages (8 tests, the important one)
test/delivery.test.mjs and test/doctor-policy.test.mjs stub gh by writing an extensionless shell script, chmod 0o755, and prepending the fixture dir to PATH joined with a hard-coded colon:
// delivery.test.mjs:149
PATH: `${fixture.dir}:${process.env.PATH}`,
// delivery.test.mjs:140, doctor-policy.test.mjs:421
chmodSync(ghPath, 0o755);
On Windows the PATH delimiter is ;, so the whole variable is corrupted rather than prepended — and even with the right delimiter, an extensionless script is not executable there. The child process therefore resolves the real gh, and the tests fire live requests:
gh: Not Found (HTTP 404)
Error: Command failed: gh api repos/acme/demo/pulls/7
Five delivery-verifier tests and the three resolver tests fail this way — including the one named "resolver integrates with deterministic GitHub fixtures", which on this platform is hitting api.github.com. If a repo named acme/demo existed with a permissive API surface, these tests would read from it. A unit suite whose stub failure mode is "talk to production GitHub with the developer's credentials" is worth closing on any platform.
Failure class 2 — POSIX permission assertions (2 tests)
login verifies /v1/me and writes config with 0600 permissions and profiles can be listed and switched assert mode 0600; NTFS has no POSIX modes, so the assertion can never hold on Windows.
Failure class 3 — the .agents/skills symlink (1 test)
init installs the method end to end dies on ENOENT … .agents\skills — this is the product-side bug already filed as #230 showing up in the suite; listed here only for completeness of the 11.
Suggested fix (small, mechanical)
- Join PATH with
path.delimiter, not":". - Write the stub as
gh.cmdon Windows (or use a tiny.mjsinvoked via agh.cmd/ghpair) — the fixture helper can emit both in one place since both test files share the pattern. - Guard the
0600assertions withprocess.platform !== "win32"(or assert viafs.statonly where modes are meaningful). - Optionally the deeper safety: have the stub tests set
GH_HOST/GH_TOKENto an invalid value so that if the stub ever fails to engage again, the suite fails fast offline instead of reaching the real API.
Happy to send the PR — I have the failing Windows environment live for verification, and it complements the Windows arc already in flight (#182, #230, #240).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with test/delivery.test.mjs and test/doctor-policy.test.mjs, then locate the shared or duplicated gh-stub setup and the 0600 permission assertions. Run node --test test/*.test.mjs in packages/cli on Windows and a POSIX system to compare the failures. Done means the stubs cannot resolve the real gh, Windows-compatible permission handling is covered, and the affected tests pass without live API requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, shell, typescript
- Domain
- cli, operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100