overengineeringstudio / overengineeringstudio/effect-utils

Implement first-class package-closure-scoped pnpm installs in shared devenv pnpm task

Open
#725 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:devenv area:nix area:tooling origin:agent type:feature
Dominant language
TypeScript
Stars
82
Forks
2
Avg merge
1d 8h
Merged PRs (30d)
121

Description

Motivation

PR schickling/schickling.dev#113 exposed a structural problem in the shared taskModules.pnpm task: consumers can list only the effect-utils packages they need, but the task still runs a workspace-root pnpm install in repos/effect-utils.

That means a downstream repo can be blocked by unrelated workspace members. In #113, schickling.dev needed packages such as:

  • @overeng/effect-rpc-tanstack
  • @overeng/notion-effect-client
  • @overeng/notion-effect-schema
  • @overeng/react-inspector
  • @overeng/utils

but the install also traversed @overeng/pty-effect -> @myobie/pty -> node-pty, whose lifecycle build required Python on the runner. That dependency was unrelated to the package closure schickling.dev needed.

The immediate schickling.dev fix used PNPM_CONFIG_IGNORE_SCRIPTS=true / npm_config_ignore_scripts=true in the effect-utils install task. That was intentionally a local unblocker, not the long-term design: install scripts should be avoided by package-closure scoping, not globally disabled as a way to hide unrelated lifecycle work.

Reproduction

From schickling.dev PR #113 context:

  1. Consume effect-utils through the megarepo checkout.
  2. Configure a nested effect-utils pnpm task with selected packages only.
  3. Run the downstream quality gate, e.g. CI=1 devenv tasks run check:all or the nested pnpm:install:effect-utils task.
  4. Observe that the task runs a whole repos/effect-utils workspace install.
  5. The install reaches unrelated packages/@overeng/pty-effect, then @myobie/pty, then node-pty, and fails when the lifecycle build cannot find the expected Python toolchain.

Current local workaround in schickling.dev:

(taskModules.pnpm {
  workspaceRoot = "repos/effect-utils";
  taskSuffix = "effect-utils";
  preInstall = ''
    export PNPM_CONFIG_IGNORE_SCRIPTS=true
    export npm_config_ignore_scripts=true
  '';
  packages = [
    "packages/@overeng/effect-rpc-tanstack"
    "packages/@overeng/notion-effect-client"
    "packages/@overeng/notion-effect-schema"
    "packages/@overeng/react-inspector"
    "packages/@overeng/utils"
  ];
})

Exploration Results

  • Adding Python would make this one lifecycle script pass, but it encodes an unrelated package's build requirements into every downstream consumer of effect-utils.
  • Passing --ignore-scripts was rejected by the shared pnpm task's purity guard, correctly: repo-local install flags should not be able to weaken the shared install contract.
  • Exporting PNPM_CONFIG_IGNORE_SCRIPTS=true in preInstall unblocked #113, but it is an escape hatch around the guard and should not become the pattern.
  • The package list already exists at the devenv module boundary, and Genie already has package-closure concepts (projectPnpmPackageClosure(...) / workspace metadata). The missing piece is making the shared pnpm task install that closure as a first-class mode.
  • E2E exploration verified that pnpm --filter <package-name>... install includes dependency closure, while leading ellipsis is the opposite direction and includes dependents.
  • A temp effect-utils manifest/lock copy with filtered frozen install selected 6 of 28 workspace projects for the schickling.dev closure. It did not run @myobie/pty / node-pty lifecycle scripts.
  • A temp fixture showed filtered install skipped an unrelated failing postinstall script, while full install failed.
  • Important pitfall: with enableGlobalVirtualStore: true, pnpm may still materialize unrelated lockfile packages under GVS links even if they are not linked into selected packages and their lifecycle scripts do not run. Scoped install narrows lifecycle blast radius; it is not a guarantee that pnpm never touches unrelated store projections.

Design Requirements

Implement an explicit package-closure-scoped install mode in nix/devenv-modules/tasks/shared/pnpm.nix.

The mode should:

  • Accept selected workspace package paths or package names and derive the full internal dependency closure.
  • Install/link only that closure, not every package in the workspace.
  • Preserve the repo-root pnpm-lock.yaml as the authoritative lockfile.
  • Preserve shared task purity defaults: frozen lockfile in CI, controlled store dir, clone-or-copy import method, side-effects cache disabled, strict store content checks, and no repo-side --ignore-scripts escape hatch.
  • Keep lifecycle scripts enabled for packages in the selected closure.
  • Avoid lifecycle scripts for packages outside the selected closure by not installing them.
  • Keep status hashing accurate: include the selected closure, relevant manifests, lockfile, pnpm workspace config, install scope, filter list, preInstall, and projection state.
  • Keep pnpm:clean scoped to node_modules projections owned by the task.
  • Produce clear diagnostics that report workspace root, selected package roots, computed closure, filters, and pnpm log location on failure.
  • Work for nested workspaceRoot values such as repos/effect-utils.
  • Continue supporting full-workspace install as the default behavior.

Potential API shape:

taskModules.pnpm {
  workspaceRoot = "repos/effect-utils";
  taskSuffix = "effect-utils";
  installScope = "packageClosure";
  packages = [
    "packages/@overeng/effect-rpc-tanstack"
    "packages/@overeng/notion-effect-client"
    "packages/@overeng/notion-effect-schema"
    "packages/@overeng/react-inspector"
    "packages/@overeng/utils"
  ];
}

Open design point: whether packages = [...] should change behavior directly, or whether a new explicit option such as installScope = "workspace" | "packageClosure" is safer. Prefer explicitness to avoid silently changing current repos.

Pitfalls / Tradeoffs

  • pnpm --filter semantics are subtle. The implementation must use --filter "${packageName}..." to include dependencies and must not accidentally use the dependent direction.
  • Hoisted bins and root-level tooling can mask missing closure members. Tests should prove package-local commands resolve without relying on unrelated root workspace packages.
  • The task currently validates node_modules projection health across root plus package paths. Scoped mode needs equivalent health checks that do not require unrelated packages to project.
  • Root node_modules may still be needed for pnpm workspace metadata. Avoid treating a minimal root projection as permission to install every workspace member.
  • dependenciesMeta.*.injected handling must stay correct; injected source paths currently influence hashing.
  • allowBuilds / ignoredBuiltDependencies / ignoreDepScripts in pnpm-workspace.yaml interact with lifecycle behavior. Scoped mode should not paper over these settings.
  • Do not solve this by globally disabling scripts, adding Python to all downstream shells, or vendoring special cases for @overeng/pty-effect.

Rollout Plan

  1. Add a scoped install option to taskModules.pnpm while keeping full-workspace install as default.
  2. Add shell/unit smoke tests for: full-workspace behavior unchanged; scoped install includes internal dependency closure; unrelated workspace lifecycle scripts do not run; lifecycle scripts inside the closure still run; impure flags remain rejected; status hash changes when selected closure changes.
  3. Add a representative fixture with packages app -> lib and unrelated native-addon-package with a failing install script.
  4. Validate in effect-utils with existing pnpm task tests.
  5. Switch schickling.dev's nested effect-utils task from the PNPM_CONFIG_IGNORE_SCRIPTS workaround to scoped mode.
  6. Run downstream validation in schickling.dev: CI=1 devenv tasks run check:all.
  7. Remove the workaround from schickling.dev once the new effect-utils input is pinned.

Acceptance Criteria

  • Downstream repos can install a selected effect-utils package closure without installing unrelated workspace packages.
  • schickling.dev no longer needs PNPM_CONFIG_IGNORE_SCRIPTS for pnpm:install:effect-utils.
  • @overeng/pty-effect -> @myobie/pty -> node-pty lifecycle scripts are not reached when no selected package depends on that closure.
  • Lifecycle scripts still execute for packages that are inside the selected closure.
  • The shared pnpm task continues to reject --ignore-scripts and other impure install flags.
  • Existing full-workspace pnpm:install behavior remains compatible.
  • Scoped install behavior has smoke coverage in nix/devenv-modules/tasks/shared/tests/.
  • CI/devenv diagnostics identify workspace root, selected packages, computed closure, filters, and pnpm log location on failure.
Posted on behalf of @schickling
field value
agent_name 🍒 co3-cherry
agent_session_id c0b70303-e0c0-447f-b9bb-f47fe1fca13e
agent_tool Codex CLI
agent_tool_version 0.131.0
agent_runtime Codex CLI 0.131.0
agent_model unknown
worktree schickling.dev/schickling-assistant/own-invoice-matching
machine dev3
tooling_profile dotfiles@4db6783

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nix/devenv-modules/tasks/shared/pnpm.nix at taskModules.pnpm and inspect the existing shared install, hashing, cleanup, and purity behavior. Then read the smoke tests under nix/devenv-modules/tasks/shared/tests/ and run the existing pnpm task tests. Done means scoped installs preserve full-workspace defaults, select dependency closures without unrelated lifecycle scripts, retain purity and hashing guarantees, and provide the requested diagnostics.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.