NVIDIA / NVIDIA/NemoClaw

perf(ci): eliminate source-loader contention and hook timeouts in CLI coverage shards

Open
#6,237 3 comments 0 reactions 1 assignee Claimed by @apurvvkumaria View on GitHub
area: ci area: e2e
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Agent Diagnostic

- Investigated CI failures using GitHub CLI (`gh`) by inspecting PR #6220, all retry attempts, and recent `CI / Main` runs.
- Reproduced the exact shard command on the PR head (`b0863c697427422e30fa6a4643f18550b30b0296`) in a clean Linux-native Node 22.23.1 image with an empty source-loader cache.
- Compared default Vitest workers with an otherwise identical `--maxWorkers=1` run.
- Inspected `vitest.config.ts`, `test/helpers/register-source-require.ts`, and the two source-heavy gateway drift suites.
- Searched open issues and PRs; no existing item tracks this failure family.

## Description

CLI coverage shards increasingly fail late in the run with Vitest's default 10-second hook timeout. The failing hook changes with shard placement, but each affected suite synchronously loads a very large TypeScript source graph through the custom CommonJS source loader.

This is increasing merge and release cycle time:

- 5 of the last 17 non-cancelled `CI / Main` runs failed with the same hook-timeout class (~29%).
- The three most recent completed main runs at investigation time failed identically.
- A failure consumes about 6m40s of shard execution and 7m41s of total CI time on average before reporting red; a blind retry pays that cost again.
- PR #6220 moved the failure family from main's shard 3 (`rebuild-gateway-drift.test.ts`) to shard 2 (`gateway-drift-preflight.test.ts`) without changing those tests or their production dependencies.

Expected behavior: source-backed coverage shards should complete reliably under the existing timeout, and suite growth or shard reshuffling should not expose infrastructure hook failures.

## Evidence

Affected runs:

- PR #6220 attempt 4: https://github.com/NVIDIA/NemoClaw/actions/runs/28633128403/job/84922896397
- Main: https://github.com/NVIDIA/NemoClaw/actions/runs/28606939333
- Main: https://github.com/NVIDIA/NemoClaw/actions/runs/28624654996
- Main: https://github.com/NVIDIA/NemoClaw/actions/runs/28631441432
- Main: https://github.com/NVIDIA/NemoClaw/actions/runs/28634921449
- Main: https://github.com/NVIDIA/NemoClaw/actions/runs/28636531834

Constrained-Linux A/B on the exact PR head, using the exact shard 2/5 coverage command and a cold cache:

| Mode | Shard wall time | Gateway first test (includes setup) | Peak cgroup memory | CPU throttling |
|---|---:|---:|---:|---:|
| Default workers | 2m12.02s | 3.187s | 1.26 GB | none |
| `--maxWorkers=1` | 2m26.22s | 0.463s | 1.03 GB | none |

Serial execution made the vulnerable setup about 6.9x faster and reduced peak memory about 18%, confirming that worker contention amplifies the cold source-loading cost. However, it made the whole shard about 11% slower, so globally serializing CLI tests would trade reliability for longer cycle time.

Neither local Linux run crossed 10 seconds. The local host is arm64 while GitHub's runner is x64, so the repeated hosted-runner failures remain the architecture-faithful reproduction. Both local modes had the same 15 unrelated Hermes permission-test failures because the minimal diagnostic image did not define the CI image's `sandbox` OS account; the gateway suite passed all assertions in both modes.

## Leading hypothesis

`test/helpers/register-source-require.ts`:

- eagerly imports TypeScript and parses the CLI tsconfig in every process;
- synchronously calls `ts.transpileModule` for every cold cache miss;
- atomically publishes cache entries, but does not coordinate simultaneous cold misses across workers;
- is installed as a setup file across the CLI project, while only 40 CLI source tests directly use `createRequire`;
- is inherited by integration child processes through `NODE_OPTIONS`.

The two repeatedly failing suites load roughly 500 source modules during their first timed `beforeEach`. Parallel workers turn that cold graph load into a resource-sensitive critical section.

This design originated in #5904 and worked when introduced, but suite growth and shard reshuffling now exceed its original headroom.

## Reproduction Steps

1. Start from a clean Linux checkout with Node 22.23.1 and no `node_modules/.cache/nemoclaw-source-require` or `node_modules/.vite` cache.
2. Install/build using the same commands as CI.
3. Run:

```shell
npx vitest run --project cli --project integration \
--shard=2/5 \
--reporter=github-actions \
--reporter=blob \
--outputFile.blob=.vitest-reports/blob-default.json \
--coverage \
--coverage.reporter=json-summary \
--coverage.reportsDirectory=coverage/cli/repro-default \
--coverage.include="bin/**/*.js" \
--coverage.include="src/**/*.ts" \
--coverage.exclude="test/**/*.js" \
--coverage.exclude="test/**/*.ts"
```

4. From another cold checkout/image, repeat with `--maxWorkers=1` and compare the first gateway test, total wall time, and peak memory.

## Proposed Work

1. Add opt-in source-loader timing/cache statistics so cold misses, duplicate transforms, transform time, and process memory are observable in CI.
2. Remove the approximately 500-module setup graphs from the two gateway suites by using native Vitest mocks or testing extracted dependency-injected boundaries.
3. Evaluate replacing eager TypeScript `transpileModule` with a lazy, source-bounded esbuild transform while preserving the current path security boundary, content-addressed cache, and atomic writes.
4. Rebalance coverage shards using observed per-file durations rather than accepting the current greater-than-2x slowest/fastest spread.
5. Add a CI reliability/performance signal for hook timeouts and slowest-shard duration.

Do not solve this with a blanket `hookTimeout` increase.

## Acceptance Criteria

- No blanket hook-timeout increase.
- Twenty repeated cold-cache runs on constrained x64 Linux complete with zero infrastructure hook timeouts.
- Gateway setup p95 stays below 5 seconds under coverage.
- All five coverage shards and merged coverage checks pass.
- Slowest-shard and total CLI gate duration do not regress; preferably they improve.
- Main-branch retry/failure rate is monitored after landing.

## Environment

- GitHub Actions: Ubuntu 24.04.4, x64, Node 22.23.1, npm 10.9.8.
- Local diagnostic: Debian Bookworm container, arm64, Node 22.23.1, 4 CPUs, 15 GiB memory, no swap.
- NemoClaw PR head: `b0863c697427422e30fa6a4643f18550b30b0296`.

## Debug Output

```shell
Error: Hook timed out in 10000ms.
If this is a long-running hook, pass a timeout value as the last argument or configure it globally with "hookTimeout".
❯ src/lib/actions/gateway-drift-preflight.test.ts:54:3
```

## Logs

See the linked GitHub Actions runs and the A/B measurements above.

## Checklist

- [x] I confirmed this bug is reproducible on GitHub-hosted runners.
- [x] I searched existing issues and this is not a duplicate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.