theam / theam/facility

test(cli): suite is POSIX-only — gh stubs never engage on Windows, so eight tests call the real GitHub API

Open
#241 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
71
Forks
64
Avg merge
15h 38m
Merged PRs (30d)
66

Description

Summary

The CLI test suite is POSIX-only in three independent ways, and the sharpest consequence is not "tests fail on Windows" — it is that eight tests silently bypass their own gh stub and call the real GitHub API from the developer's machine.

Run context: Windows 11, Node 22.23.2, pnpm 11.20.0 (stable, no flags), clean main, pnpm install --frozen-lockfile green. node --test test/*.test.mjs in packages/cli: 86 pass / 11 fail.

Failure class 1 — the gh stub never engages (8 tests, the important one)

test/delivery.test.mjs and test/doctor-policy.test.mjs stub gh by writing an extensionless shell script, chmod 0o755, and prepending the fixture dir to PATH joined with a hard-coded colon:

// delivery.test.mjs:149
PATH: `${fixture.dir}:${process.env.PATH}`,
// delivery.test.mjs:140, doctor-policy.test.mjs:421
chmodSync(ghPath, 0o755);

On Windows the PATH delimiter is ;, so the whole variable is corrupted rather than prepended — and even with the right delimiter, an extensionless script is not executable there. The child process therefore resolves the real gh, and the tests fire live requests:

gh: Not Found (HTTP 404)
Error: Command failed: gh api repos/acme/demo/pulls/7

Five delivery-verifier tests and the three resolver tests fail this way — including the one named "resolver integrates with deterministic GitHub fixtures", which on this platform is hitting api.github.com. If a repo named acme/demo existed with a permissive API surface, these tests would read from it. A unit suite whose stub failure mode is "talk to production GitHub with the developer's credentials" is worth closing on any platform.

Failure class 2 — POSIX permission assertions (2 tests)

login verifies /v1/me and writes config with 0600 permissions and profiles can be listed and switched assert mode 0600; NTFS has no POSIX modes, so the assertion can never hold on Windows.

Failure class 3 — the .agents/skills symlink (1 test)

init installs the method end to end dies on ENOENT … .agents\skills — this is the product-side bug already filed as #230 showing up in the suite; listed here only for completeness of the 11.

Suggested fix (small, mechanical)

  • Join PATH with path.delimiter, not ":".
  • Write the stub as gh.cmd on Windows (or use a tiny .mjs invoked via a gh.cmd/gh pair) — the fixture helper can emit both in one place since both test files share the pattern.
  • Guard the 0600 assertions with process.platform !== "win32" (or assert via fs.stat only where modes are meaningful).
  • Optionally the deeper safety: have the stub tests set GH_HOST/GH_TOKEN to an invalid value so that if the stub ever fails to engage again, the suite fails fast offline instead of reaching the real API.

Happy to send the PR — I have the failing Windows environment live for verification, and it complements the Windows arc already in flight (#182, #230, #240).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test/delivery.test.mjs and test/doctor-policy.test.mjs, then locate the shared or duplicated gh-stub setup and the 0600 permission assertions. Run node --test test/*.test.mjs in packages/cli on Windows and a POSIX system to compare the failures. Done means the stubs cannot resolve the real gh, Windows-compatible permission handling is covered, and the affected tests pass without live API requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, shell, typescript
Domain
cli, operating-systems, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.