garrytan / garrytan/gstack

test:free runs the real `gbrain` when it is on PATH; brain-cache tests hit a remote brain and trip the 30s exec timeout

Open
#2,829 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
133k
Forks
19.9k
Avg merge
18h 46m
Merged PRs (30d)
26

Description

## Summary

`bun run test:free` is not hermetic on a machine that has `gbrain` installed and pointed at a remote (Supabase) brain. Several tests reach `lib/gbrain-exec.ts` through the real binary on PATH, so each call is a network round-trip (~1.5 s here). Under the free-suite shard load, one brain-cache test crossed the 30 s default exec timeout and failed. gstack's own CI never installs gbrain, so the suite is green upstream and the failure only shows up on a user's machine — in my case inside a launchd auto-updater that runs `./setup && bun run test:free` and rolls back on red.

## Environment

- gstack 1.81.0.0 (`0530392`), macOS 15, bun via `~/.bun/bin`
- gbrain 0.48.x installed globally, configured against a remote Supabase brain (not PGLite)
- Suite run unattended (launchd), 6 shards

## Observed

```
(fail) brain-cache malformed _meta.json (#1879) > cmdGet does not throw on null / array / primitive _meta.json [30759.31ms]
```

`test/brain-cache-roundtrip.test.ts` writes a malformed `_meta.json` under a temp `HOME`, so every entity is stale; `cmdGet` in `bin/gstack-brain-cache` then refreshes via `execGbrainJson` / `spawnGbrain` (`lib/gbrain-exec.ts`, `timeout: opts.timeout ?? 30_000`). The temp `HOME` isolates the cache dir but not the binary: `gbrain` still resolves from PATH and talks to the user's real brain. Four such lookups at ~1.5 s each fit comfortably in isolation but not under shard contention, and the whole test is then reported as a failure of the assertion under test rather than of the environment.

Runs of the same suite with `gbrain` removed from PATH pass every time (`GSTACK_FREE_RETRY_FLAKY=1` did not rescue the red run because the failure reproduces serially once the brain is slow).

## What does *not* work as a workaround

A stub `gbrain` that exits non-zero first on PATH is not equivalent to "absent": `test/gbrain-sync-voyage-code-3-integration.test.ts` gates on `which gbrain` (plus `VOYAGE_API_KEY`) and then fails 3 tests against the stub, reproducibly. So the only green state on a dev machine with gbrain installed is to build a PATH without it, which is what I ended up doing in my updater (mirror every PATH dir that contains gbrain into exec-wrappers minus gbrain, run the suite on that).

## Suggested fix (either would do)

1. In the brain-cache tests, stub the exec layer (`lib/gbrain-exec.ts`) or set whatever env makes `bin/gstack-brain-cache` treat gbrain as unreachable, so `cmdGet` exercises the malformed-meta path without a real spawn. Those tests are about `_meta.json` handling, not about gbrain.
2. More generally, have `test:free` refuse to spawn a real `gbrain` unless an explicit opt-in (e.g. `GSTACK_TEST_REAL_GBRAIN=1`) is set, mirroring how the voyage integration test already gates on env. Then a developer's installed gbrain can't leak into the "free" suite.

Happy to send a PR for (1) if you'd take it.

Contributor guide

Open the contributing guide

Research direction

Start with test/brain-cache-roundtrip.test.ts and run the malformed _meta.json case under bun run test:free. Trace bin/gstack-brain-cache through lib/gbrain-exec.ts to see where the real gbrain spawn occurs. Done means these cache tests exercise malformed metadata without a remote brain or 30-second timeout, while the existing voyage integration gating remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
bun, typescript
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.