overengineeringstudio / overengineeringstudio/effect-utils

mkPnpmCli: eliminate IFD to support cross-system flake evaluation

Open
#246 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:nix origin:agent type:chore type:feature
Dominant language
TypeScript
Stars
82
Forks
2
Avg merge
1d 8h
Merged PRs (30d)
121

Description

Problem

mk-pnpm-cli.nix uses Import From Derivation (IFD) — it calls builtins.readFile and lib.cleanSourceWith on the workspaceRoot argument during Nix evaluation. When workspaceRoot is a derivation (e.g. from pkgs.runCommand), this forces the derivation to be built at eval time.

This breaks cross-system evaluation. For example, when an aarch64-darwin machine evaluates x86_64-linux outputs (via eachDefaultSystem), it can't build the system-specific runCommand derivation locally. nix flake check and nix eval fail without remote builders.

IFD locations in mk-pnpm-cli.nix
  • Line 74: builtins.readFile (workspaceRootPath + "/${packageDir}/pnpm-workspace.yaml")
  • Line 306: builtins.fromJSON (builtins.readFile (workspaceRootPath + "/${packageDir}/package.json"))
  • Lines 177-178, 275-278: lib.cleanSourceWith { src = workspaceRootPath; ... }
Why consumers use derivations as workspaceRoot

When a CLI package lives in a different repo structure than its workspace dependencies (e.g. a dotfiles repo consuming @overeng/* packages from effect-utils), consumers build a combinedWorkspace derivation that:

  1. Mounts the main package and external deps into a unified directory tree
  2. Rewrites pnpm-workspace.yaml to match the new layout
  3. Patches pnpm-lock.yaml importers/specifiers with sed/yq (fragile)
  4. Patches package.json to fix paths

This is 50-100 lines of brittle runCommand boilerplate per consumer (6 consumers in dotfiles alone).

Proposed solution: internalize workspace assembly

Replace the current API where consumers pre-build a workspace derivation with a new API where mkPnpmCli accepts individual source ingredients and assembles the workspace itself at build time (not eval time).

Key insight

Mount workspace deps at positions matching the lockfile's relative paths. The lockfile stays untouched — eliminating all sed/yq rewriting.

Current API
mkPnpmCli {
  name = "otel";
  workspaceRoot = combinedWorkspace;  # ← derivation, triggers IFD
  packageDir = "flakes/otel-cli";
  entry = "flakes/otel-cli/bin/otel.ts";
  pnpmDepsHash = "sha256-...";
}
Proposed API
mkPnpmCli {
  name = "otel";
  packageSource = ./.;                # plain Nix path — no IFD
  entry = "bin/otel.ts";             # relative to package root

  # Workspace members: mounted at lockfile-relative positions
  workspaceDeps = {
    "../tui-core"  = effectUtilsSrc + "/packages/@overeng/tui-core";
    "../tui-react" = effectUtilsSrc + "/packages/@overeng/tui-react";
    "../utils"     = effectUtilsSrc + "/packages/@overeng/utils";
    "../utils-dev" = effectUtilsSrc + "/packages/@overeng/utils-dev";
  };

  # Deps mounted at path but NOT added to workspace yaml
  linkDeps = {};

  # jq filter applied to package.json at build time
  packageJsonFilter = ''del(."$genie")'';

  patchesDir = "../utils/patches";
  pnpmDepsHash = "sha256-...";
}
Consumer patterns identified
Pattern Consumers workspaceDeps linkDeps
Simple (no external deps) bird, gh-ci-exporter {} {}
Link dep only op-secret-cache, agent-exporter {} { "../utils" = ... }
Workspace members otel-cli 4 members {}
Complex hybrid oi ~10 members { geist = ... }
Benefits
  • No IFD: All eval-time reads (builtins.readFile, cleanSourceWith) operate on plain paths/flake inputs
  • No lockfile patching: Mounting at lockfile-relative paths means zero sed/yq rewriting
  • Less boilerplate: ~80 lines of combinedWorkspace per consumer → ~10 lines of workspaceDeps
  • Cross-system eval: nix flake check works without remote builders
Open questions
  1. Should packageJsonFilter be a jq expression string, or a Nix function?
  2. How to handle oi's catalog: patching in livestore deps — prePatch script param vs overrideAttrs?
  3. Should the builder preserve non-packages fields from the original pnpm-workspace.yaml (e.g. supportedArchitectures, publicHoistPattern)?

Context: dotfiles#254

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mk-pnpm-cli.nix, especially the IFD locations at lines 74, 177-178, 275-278, and 306, then review the listed consumer patterns and the dotfiles#254 context. Use nix flake check and nix eval to verify cross-system evaluation. Done means workspace assembly occurs at build time, the lockfile is not patched, and the proposed API handles the documented consumer cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
build-system
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.