overengineeringstudio / overengineeringstudio/effect-utils

CI: cold-install pnpm-state flake on Namespace runners amplified by success()-gated cache-save

Open
#917 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
82
Forks
2
Avg merge
1d 8h
Merged PRs (30d)
121

Description

Downstream consumers on Namespace cloud runners intermittently fail CI on lock-changing PRs with a cold-install node_modules/.bin materialization failure (Executable not found in $PATH: vitest/madge), preceded by a determinate-nix daemon self-test error (cannot connect to socket at /nix/var/nix/daemon-socket/socket). Observed on livestorejs/livestore#1397 (effect-utils bump). Same commit reruns rotate which jobs pass/fail → confirmed runner-health flake, not code.

Why it sticks (amplifier)

The generated per-job Save pnpm state step is gated:

if: ${{ success() && steps.restore-pnpm-state.outputs.cache-hit != 'true' }}

On a lock change every job cold-misses the pnpm-state cache. success() = whole-job success, so any job whose install completes but whose later step flakes (or whose daemon self-test fails) never seeds the cache → subsequent reruns keep cold-missing → the flake-prone cold path runs every time until a fully-green run happens by luck.

Two improvements (durable, careful — not urgent)

  1. Safe cache seeding: seed the pnpm-state cache when the install itself verifiably completed (node_modules/.bin populated, .modules.yaml present, install-state/projection hashes written) even if a later job step fails — but NOT when the install is incomplete (avoid caching a broken links/). This raises seeding probability → fewer flake-prone cold installs on rerun, without caching corruption. Requires a discrete verify-install step to gate on instead of whole-job success().
  2. Daemon/setup retry robustness: the existing run-with-nix-gc-race-retry.sh wraps task runs, not the Nix-daemon self-test / devenv setup phase. Wrap (or retry) the daemon-setup/install phase so a transient daemon-socket failure retries instead of failing the whole job.

Root cause of the flake itself is Namespace/determinate-nix daemon availability (external infra); these two changes reduce its blast radius on downstream CI. Refs: pnpm.nix cold-path ~L455-475 / L575-632; genie CI generator Save pnpm state step. Filed from the livestore#1397 investigation; relates to the effect-utils consumption/prevention story (frequent-bump + canary).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pnpm.nix around the referenced cold-path lines and the genie CI generator's Save pnpm state step; read how the current install and cache-hit conditions are produced. Inspect run-with-nix-gc-race-retry.sh and the devenv setup path for the daemon self-test. Done means incomplete installs cannot seed pnpm state, completed installs can seed it after later failures, and transient daemon-setup failures retry.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, shell, typescript
Domain
build-system, ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.