overengineeringstudio / overengineeringstudio/effect-utils

Epic: hermetic dependency artifacts for pure cross-worktree reuse

Open
#946 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:devenv area:nix type:epic
Dominant language
TypeScript
Stars
82
Forks
2
Avg merge
1d 8h
Merged PRs (30d)
121

Description

Problem

PR #922 fixes dependency-edge authority and retains major cross-worktree byte
reuse, but its current whole pnpm Store Cache is not the long-term purity
maximum. v11/index.db is shared writable SQLite state whose identity is the
ambient union of installs. It is disposable and non-authoritative, but it is
neither declared-input-addressed nor immutable.

The accepted dependency-materialization contract now makes the distinction
explicit:

  • only declared-input-derived, integrity/content-addressed, immutable state is
    eligible for cross-root reuse;
  • current GVS is inadmissible because it shares mutable topology/repair state;
  • whole-Store-Cache index sharing is a transitional implementation delta;
  • root-local mutable realization is a compatibility boundary, not the target.

The earlier split-v11/files design is not a solution: independent indexes did
not discover the shared data effectively (12 reused / 600 downloaded in the
real two-root experiment), and native profile-local prune deleted data required
by sibling roots.

Outcome

Make the reusable unit a Hermetic Dependency Artifact keyed by the complete
Materialization Profile and consumed read-only across every compatible root.
Only root-specific workspace projection, mutable lock-update work, and explicit
native leaves remain root-local.

Use an immutable, atomically published Store Cache seed with root-local writable
indexes/overlays only as a stepping stone when it directly reuses the existing
prepared-deps producer and converges on the same artifact identity. Do not
create a second cache/profile vocabulary or synthesize pnpm's private SQLite
schema.

Hard constraints

  • Lifecycle scripts, ambient downloads during consumption, side-effect caches,
    mutable native/build output, and consumer mutation are forbidden.
  • Identity includes lockfile, workspace manifests/members, catalogs, patches,
    pnpm/Node versions, install policy, platform/optional/native classification,
    and every other topology-affecting input.
  • Publication is atomic; consumers observe an old complete artifact or a new
    complete artifact, never a partial one.
  • Cross-root reuse does not grant graph, mutation, repair, prune, or GC
    authority.
  • Eviction/corruption of one root's writable state cannot affect another root or
    the reusable artifact.
  • Linux hardlinks alone are not an immutability boundary: attempted writes
    through every imported alias must fail or leave the artifact hash unchanged.
  • GC is artifact-owner reachability, not mutually invisible pnpm indexes over a
    shared files pool.

Architecture

declared Materialization Profile
  -> private lifecycle-free producer
  -> validate + normalize + hash
  -> atomic immutable publication
  -> broad read-only reuse
       -> root A workspace/projection state
       -> root B workspace/projection state
       -> CI/Nix consumers

Reuse the existing mk-pnpm-deps.nix prepared-dependency/FOD and profile model
as the starting producer. pnpm remains the resolver and lockfile writer; the
artifact path is a faithful materialization consumer, not a second resolver.

Required experiments and gates

  1. Identity completeness: perturb every semantic input; each relevant
    perturbation changes the artifact key, while source-only edits do not.
  2. Reproducibility: two clean private builds per supported system are
    byte-identical; negative fixtures prove zero lifecycle execution.
  3. Atomic publication: kill the producer at every phase; no consumer can
    observe a partial artifact. Corrupt artifacts fail integrity validation.
  4. Network denial: a second absent root consumes the artifact with transport
    denied and records zero connections/bytes, not merely reporter counters.
  5. Mutation isolation: root A install/update/prune/corrupt/repair leaves the
    artifact and root B byte-identical.
  6. Concurrency: two cold roots consume one artifact without a shared
    writable DB or cross-root mutex; crash injection cannot poison siblings.
  7. Physical reuse: report logical/allocated/unique bytes, files/inodes, and
    shared-block ratio on Linux/ext4 and Darwin/APFS.
  8. Latency/work: compare cold, second-root, warm, offline, repair, and
    concurrent phases against #922's whole-store baseline and isolated stores.
  9. Recovery/GC: independently evict/rebuild a root overlay and the artifact;
    reclaim one artifact generation without coordinating unrelated consumers.
  10. Live compatibility: prove workspace live-source links, lock mutation,
    projection freshness, exact native-leaf replacement, and prepared CLI/Nix
    consumers.

Benchmark baseline

Current #922 effect-utils/ext4 measurements (629 packages):

realization mean second-root reuse shared blocks
split files + clone/copy 24.7015 s 12 reused / 600 downloaded 0.7%
root-local index + hardlink 15.056 s 12 reused / 600 downloaded 89.6%
whole shared Store Cache 6.8815 s 612 reused / 0 downloaded 89.6%

The new design must satisfy purity/authority gates first, then seek a
non-dominated point among evaluated admissible candidates. A speed or byte win
cannot legalize shared mutation.

Delivery

  • Record the current whole-store path only as the compatibility baseline and
    keep DMP DELTA-001 open until code converges.
  • Prototype immutable seed + root-local writable overlay without production
    adoption; falsify it if pnpm cannot consume it safely.
  • Measure direct live consumption of the existing prepared-dependency
    artifact before inventing a new producer.
  • Select seed stepping stone or direct Hermetic Dependency Artifact through
    the hard gates and same-workload benchmark matrix.
  • Implement one authoritative producer and root-local projection consumer.
  • Add typed CI scenarios under #945 for identity, completeness, correctness,
    reuse, storage, latency, concurrency, and recovery.
  • Prove effect-utils, dotfiles/Vista, and one simpler downstream consumer on
    Linux and Darwin.
  • Remove whole-store shared-index leases/prune/migration behavior and delete
    DMP DELTA-001 plus fleet DELTA-001.
  • Update PR #922 and downstream merge contracts so no artifact claims the
    compatibility realization is the long-term architecture.

Related and ownership

  • #922 — current compatibility implementation and measured baseline
  • #945 — typed measurement scenarios and truthful completeness
  • #353 — canonical workspace topology planner and shared identity inputs
  • schickling/dotfiles#502 — earlier Nix-centric lockfile-consumer exploration;
    this issue narrows ownership to effect-utils and integrates the live-worktree
    path
  • schickling/dotfiles#1306 — fleet consumer and native projection

effect-utils owns the artifact/profile/materialization semantics. Consumers own
their workload declarations and root-specific projections; fleet owns placement
of immutable artifacts and local writable overlays, not dependency identity.

Posted on behalf of @schickling
field value
agent_name 🦜 co3-macaw
agent_session_id 2704af84-dabe-4279-99fb-e8e7c3920c7a
agent_tool Codex CLI
agent_tool_version 0.144.1
agent_runtime Codex CLI 0.144.1
agent_model unknown
runtime_profile /nix/store/qq3avrif77r90ypqbdv3hgd3gvwj5s32-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest /nix/store/pjlb3cwf453ghzmc8jj4v8h29sw6p4dg-agent-skills-corpus/share/agent-skills/manifest.json
worktree effect-utils/schickling-assistant/2026-07-17-pnpm-projection-authority
machine dev3
tooling_profile dotfiles@4b8e1c5

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing mk-pnpm-deps.nix prepared-dependency/FOD and profile model, then review the compatibility baseline in #922 and typed scenarios planned under #945. Compare the proposed producer and root-local projection against the listed identity, atomicity, network, mutation, concurrency, reuse, and recovery gates. Done means one authoritative implementation passes the required scenarios for the named consumers on Linux and Darwin.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.