Windows: 1435 of 7126 tests fail on a fresh clone of main, so tier 1 gives contributors no signal
- Dominant language
- TypeScript
- Stars
- 133k
- Forks
- 19.9k
- Avg merge
- 18h 46m
- Merged PRs (30d)
- 26
Description
## Summary
On a fresh clone on Windows, `bun test` reports **1435 failures out of 7126 tests** before any change is made. CONTRIBUTING treats tier 1 as the gate for every PR, so on Windows there is effectively no usable signal: a contributor cannot tell whether their change broke something or whether it was already red.
The causes look environmental rather than genuine product bugs, and two of the three look cheap to fix.
## Environment
| | |
|---|---|
| OS | Windows 11 Pro 10.0.26200 (build 26200) |
| gstack | `main` @ v1.66.0.0 |
| bun | 1.3.14 |
| Node | v24.19.0 |
| Shell | Git Bash (MINGW64) |
Fresh `git clone`, `bun install`, `bun test`. No local modifications.
```
5241 pass
450 skip
1435 fail
12 errors
19265 expect() calls
Ran 7126 tests across 507 files. [292.84s]
```
Reproduced identically on three separate runs.
## Cause 1: extensionless shell scripts invoked via `execSync`
The largest cluster. Tests call `bin/` scripts directly:
```
error: Command failed: ...\gstack-pr\bin/gstack-builder-profile
'...\gstack-pr\bin/gstack-builder-profile' is not recognized as an
internal or external command, operable program or batch file.
at execSync (node:child_process:280:14)
at runProfile (test\builder-profile.test.ts:19:18)
```
`bin/gstack-builder-profile` is an extensionless bash script. `execSync` routes through `cmd.exe`, which has no shebang handling and no `PATHEXT` entry that matches, so it cannot execute it. This is independent of Git Bash being installed, since the test does not go through it.
Affected suites include (fail counts from one run):
| Suite | Fails |
|---|---|
| `gstack-builder-profile` | 13 |
| `gstack-model-benchmark --dry-run` | 12 |
| `gstack-brain-sync secret scan` | 8 |
| `gstack-brain-enqueue` | 6 |
| `gstack-model-benchmark prompt resolution` | 5 |
| `gstack-config gbrain keys` | 5 |
| `init + sync + restore round-trip` | 4 |
**Possible fix:** have the test helpers prefix with `bash` on Windows, e.g. `execSync(process.platform === 'win32' ? \`bash "${script}"\` : script)`. That keeps the scripts themselves untouched.
## Cause 2: `jq` is not installed
Migration tests shell out to `jq`:
```
(fail) v1.38.1.0 migration > adds entries to privacy-map.json via jq (preserves JSON validity)
(fail) v1.38.1.0 migration > adds patterns to allowlist before USER ADDITIONS marker
(fail) v1.40.0.0 migration > ...
```
`jq` is not part of Git for Windows and is not in `package.json`, so a Windows contributor following CONTRIBUTING will not have it. Same for `sqlite3`.
On this machine:
```
jq: MISSING
sqlite3: MISSING
bash, perl, sed, awk, git: present (Git Bash)
```
**Possible fix:** skip these tests when `jq` is absent rather than failing them, and note the dependency in CONTRIBUTING. Bun ships an embedded SQLite, so the `sqlite3` CLI dependency may be avoidable too.
## Cause 3: unbuilt binaries
Some suites correctly skip:
```
[skip] make-pdf binary missing (...\make-pdf\dist\pdf). Run bun run build.
```
That pattern works well. Others in the same situation appear to fail instead. Extending the existing skip guard to those would remove more noise.
## Why this matters
CONTRIBUTING says tier 1 "runs on every commit, <5s" and is the gate before merging. On Windows it takes ~5 minutes and returns 1435 failures on unmodified `main`, so the only way a contributor can use it is to diff failure counts against a baseline run, which is not obvious and is not documented.
I hit this while preparing #2596. I could only claim that PR was test-neutral by running the suite twice and comparing counts (identical on both: 5241 / 450 / 1435 / 12).
## Offer
Happy to open a PR for cause 1 and cause 2 if that direction sounds right. Cause 1 looks like a change to the test helpers rather than to any shipped script, and cause 2 looks like a skip guard plus a CONTRIBUTING note. I would want a maintainer to confirm the approach before I write it, since the fix touches shared test infrastructure.
I can also attach the full 1.3 MB test log if useful.
Contributor guide
Research direction
Start by running `bun test` on Windows and reviewing `test/builder-profile.test.ts`, the other affected suites, and the `bin/` scripts they invoke. Check the existing skip guard for the missing `make-pdf` binary and the dependency guidance in `CONTRIBUTING`. Done means a fresh Windows clone no longer reports these environmental failures, or clearly skips them with documented prerequisites while preserving meaningful test failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, typescript
- Domain
- operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100