test:free runs the real `gbrain` when it is on PATH; brain-cache tests hit a remote brain and trip the 30s exec timeout
- Dominant language
- TypeScript
- Stars
- 133k
- Forks
- 19.9k
- Avg merge
- 18h 46m
- Merged PRs (30d)
- 26
Description
## Summary
`bun run test:free` is not hermetic on a machine that has `gbrain` installed and pointed at a remote (Supabase) brain. Several tests reach `lib/gbrain-exec.ts` through the real binary on PATH, so each call is a network round-trip (~1.5 s here). Under the free-suite shard load, one brain-cache test crossed the 30 s default exec timeout and failed. gstack's own CI never installs gbrain, so the suite is green upstream and the failure only shows up on a user's machine — in my case inside a launchd auto-updater that runs `./setup && bun run test:free` and rolls back on red.
## Environment
- gstack 1.81.0.0 (`0530392`), macOS 15, bun via `~/.bun/bin`
- gbrain 0.48.x installed globally, configured against a remote Supabase brain (not PGLite)
- Suite run unattended (launchd), 6 shards
## Observed
```
(fail) brain-cache malformed _meta.json (#1879) > cmdGet does not throw on null / array / primitive _meta.json [30759.31ms]
```
`test/brain-cache-roundtrip.test.ts` writes a malformed `_meta.json` under a temp `HOME`, so every entity is stale; `cmdGet` in `bin/gstack-brain-cache` then refreshes via `execGbrainJson` / `spawnGbrain` (`lib/gbrain-exec.ts`, `timeout: opts.timeout ?? 30_000`). The temp `HOME` isolates the cache dir but not the binary: `gbrain` still resolves from PATH and talks to the user's real brain. Four such lookups at ~1.5 s each fit comfortably in isolation but not under shard contention, and the whole test is then reported as a failure of the assertion under test rather than of the environment.
Runs of the same suite with `gbrain` removed from PATH pass every time (`GSTACK_FREE_RETRY_FLAKY=1` did not rescue the red run because the failure reproduces serially once the brain is slow).
## What does *not* work as a workaround
A stub `gbrain` that exits non-zero first on PATH is not equivalent to "absent": `test/gbrain-sync-voyage-code-3-integration.test.ts` gates on `which gbrain` (plus `VOYAGE_API_KEY`) and then fails 3 tests against the stub, reproducibly. So the only green state on a dev machine with gbrain installed is to build a PATH without it, which is what I ended up doing in my updater (mirror every PATH dir that contains gbrain into exec-wrappers minus gbrain, run the suite on that).
## Suggested fix (either would do)
1. In the brain-cache tests, stub the exec layer (`lib/gbrain-exec.ts`) or set whatever env makes `bin/gstack-brain-cache` treat gbrain as unreachable, so `cmdGet` exercises the malformed-meta path without a real spawn. Those tests are about `_meta.json` handling, not about gbrain.
2. More generally, have `test:free` refuse to spawn a real `gbrain` unless an explicit opt-in (e.g. `GSTACK_TEST_REAL_GBRAIN=1`) is set, mirroring how the voyage integration test already gates on env. Then a developer's installed gbrain can't leak into the "free" suite.
Happy to send a PR for (1) if you'd take it.
Contributor guide
Research direction
Start with test/brain-cache-roundtrip.test.ts and run the malformed _meta.json case under bun run test:free. Trace bin/gstack-brain-cache through lib/gbrain-exec.ts to see where the real gbrain spawn occurs. Done means these cache tests exercise malformed metadata without a remote brain or 30-second timeout, while the existing voyage integration gating remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bun, typescript
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100