Genie: complete projection artifacts into a cross-authority committed-contract system

Current status (2026-08-11)

• projectionArtifact.json() landed in #755; no implementation PR is open.

• The next evidence milestone is one bounded API/prototype comparing an additive projectionArtifact.json() extension with a thin contract wrapper.

• Downstream correctness fixes and domain migrations must remain migration-ready but must not block on this generic extraction.

Ownership boundary

• Owns the generic contract envelope, canonical bytes, provenance/fingerprints, atomic check/publication, and cross-language consumer recipes.

• Does not own Weaver semantics or renderers, the otel-scrape migration (#882), or consumer-specific catalogs and policy.

Milestone shape

• M0: settle the API with a bounded prototype and non-TypeScript producer proof.

• M1: implement the additive committed-projection contract profile.

• M2: prove Nix, Rust, TypeScript/Effect, and compiled Bun consumers reject unsupported schemas.

• M3: migrate two real adopters and delete equivalent local plumbing.

Problem

Across the composed repositories we repeatedly need the same boundary:

authoritative typed/Nix/Rust/remote contract
  -> validate domain invariants
  -> deterministic schema-versioned committed projection
  -> Nix / TypeScript / Bun / Rust consumers
  -> provenance + semantic-input identity + freshness proof

The system already has most of the right pieces, but they are assembled independently by each projection family. The resulting artifacts differ on generated markers, schema versions, fingerprints, atomic/read-only publication, consumer decoding, and CI freshness. A new projection often starts by hand-rolling stable sorting, hashing, --check, and error reporting again.

This is not a request for a second generator framework. @overeng/genie already owns the correct base mechanisms:

• projectionArtifact.json() keeps typed source data, projects it, adds schemaVersion, sorts recursively, and runs reusable validators.
• Genie owns deterministic write/check behavior and read-only generated outputs.
• The Weaver registry demonstrates one typed authority projecting to YAML, TypeScript, and Rust with target-specific semantic fingerprints and provenance.

The missing piece is a reusable committed projection contract profile that composes these mechanics for TypeScript-authored and non-TypeScript authorities, without moving ecosystem-specific semantics into Genie.

Survey evidence

A read-only survey covered all eight members of schickling/megarepo-all and inspected 440 Genie source/output pairs plus custom codegen families. Public examples include:

Existing reusable core in effect-utils

• packages/@overeng/genie/src/runtime/projection-artifact/mod.ts
  • typed .data
  • pure project
  • required schemaVersion
  • canonical recursive key ordering
  • validation hooks and uniqueValues
• pnpm-install-contract.json.genie.ts
  • real committed projection consumed by shell/Nix machinery
  • currently has a schema version and Genie freshness, but no standard embedded semantic-input fingerprint/provenance envelope
• genie/weaver-registry/registry.ts and genie/weaver-registry/*
  • 16 typed contract seams -> 5 committed YAML/TS/Rust targets
  • split doc, TS identity, and Rust identity fingerprints
  • provenance headers, uniqueness/reference validation, no-orphan checks, freshness, and external Weaver validation

Repeated public downstream implementations

a downstream infrastructure repository contains six strong independent projection families with 21 explicitly named committed artifacts:

1. SCG typed policy/catalog -> versioned JSON/docs -> Nix and TypeScript consumers
2. pinned CLI grammars -> eight TS snapshots plus one manifest
3. Rust serde/schemars contract -> JSON Schema, samples, and generated TypeScript
4. Effect/TypeScript feedback contract -> committed JSON -> Rust build.rs
5. typed telemetry registry -> TypeScript and Rust constants
6. Effect-schema inventory -> JSON/TOML operational projections

Among those 21 artifacts, 13 have a generated marker, 5 carry schema/version data, and none is generator-enforced read-only. Each family implements a different subset of hashing, provenance, freshness, and CI wiring.

Other public examples include Rust/OpenAPI/GraphQL/remote-schema/binary-asset projections. They confirm that the generic nucleus is common, while their extraction and rendering semantics remain ecosystem-specific.

Existing work and duplicate check

• PR #755 added projectionArtifact.json() and is the foundation for this issue.
• #916 concerns Rust <-> rich Effect Schema semantic interoperability. It is adjacent but distinct: this issue concerns deterministic committed projection transport and integrity, not a cross-language schema IDL.
• #831 concerns separating Genie's codegen runtime from its TUI dependency surface. Any reusable projection API should remain compatible with that split.
• #574 concerns task-cache invalidation for .genie.ts inputs. It affects freshness reliability but does not define the projection contract.
• #882 is a domain migration onto the Weaver generator, not a generic projection mechanism.

No existing issue found defines the cross-authority committed projection profile below.

Recommended shape

Extend Genie through composable layers rather than introducing a universal IDL or a parallel generator.

1. Pure projection-contract primitives

Build on projectionArtifact.json() with an opt-in contract envelope and helpers for:

• stable canonical JSON bytes
• explicit contract ID and schema version
• generator/profile version
• declared semantic source paths/identifiers
• deterministic semantic-input fingerprinting
• regeneration/check command metadata
• generic uniqueness and referential-integrity validators
• structured validation/freshness errors

The fingerprint input must be explicit and semantic. It must not include wall-clock time, $PWD, ambient environment, secrets, or irrelevant documentation unless that target intentionally projects them.

A JSON artifact needs valid embedded provenance because JSON cannot carry Genie's comment header. The exact reserved field name is a design decision; the required information is not:

{
  "schemaVersion": 1,
  "_generated": {
    "contract": "example/display-codes",
    "generator": "genie-projection-contract/v1",
    "sources": ["path/to/authority"],
    "regenerate": "dt projection:generate",
    "semanticInputFingerprint": "sha256:..."
  },
  "data": {}
}

2. Target families, not one global fingerprint

Allow related targets to share validated source data while declaring target-specific semantic input scopes. The Weaver registry is the model: a documentation-only edit must not churn Rust identity constants, while a Rust signal-name change must.

Renderers remain small and explicit. Generic core should provide normalized data, fingerprints, provenance, and validation; domain adapters own JSON/YAML/TS/Rust rendering details.

3. Non-TypeScript producer adapter

Prototype one thin materialize/check adapter for deterministic producers such as Nix or Rust:

producer emits canonical source projection/spec
  -> Genie projection contract validates/envelopes it
  -> atomic target publication or byte-for-byte check

This may be a lightweight Genie CLI mode or task helper. It must not require consumers to reimplement a Nix or Rust registry in TypeScript. Nix remains authoritative when Nix owns the source; Rust remains authoritative when Rust owns the wire model.

4. Consumer recipes, not a runtime service

Document and test small fail-closed consumers:

• Nix: tracked file + builtins.fromJSON, assert contract/schema version
• Rust: include_str! + serde decoder, reject unsupported schema
• TypeScript/Effect: static JSON import + Effect Schema decoder
• compiled Bun: static import/embedding smoke test

Unknown domain values are a consumer policy. Invalid artifacts or unsupported schema versions are contract failures.

Invariants

1. Exactly one declared authority produces each artifact.
2. Projection is deterministic and canonically serialized with a trailing newline.
3. Every committed machine-readable projection declares contract identity, schema version, generator identity, semantic sources, regeneration command, and semantic-input fingerprint.
4. Fingerprints cover all and only semantic inputs for that target.
5. Generation is atomic and restores the repository's read-only generated-file policy.
6. Check mode is non-mutating and compares expected bytes, metadata, and permissions.
7. Local and CI freshness use the same command and non-empty target set.
8. Missing, malformed, stale, unsupported, or manually edited artifacts fail loudly.
9. Domain-specific validation stays with the owning registry/package.
10. Store-only build intermediates remain Nix outputs; only intentional bootstrap/review/runtime projections are committed.

Proposed API exploration

Keep names provisional, but compare an incremental extension against a small contract wrapper:

projectionContract.json({
  contract: 'example/display-codes',
  schemaVersion: 1,
  generatorVersion: 1,
  sources: ['nixpkgs/ai/personas.nix'],
  regenerate: 'dt projection:generate',
  data,
  semanticInputs: (data) => data,
  project: (data) => ({ personas: data.personas }),
  validators: [...],
})

For multi-target families, reuse the validated data but select semantic inputs per target instead of forcing one omnibus hash.

Test strategy

Core behavior

• permutation-equivalent objects emit identical bytes
• unchanged semantic input leaves the target byte-identical
• relevant input changes update the fingerprint and content
• irrelevant input changes do not churn a narrower target
• schema/generator version changes are explicit
• no current time, ambient environment, or absolute working path enters output
• duplicate/referential-integrity validators return structured issues
• invalid top-level or non-JSON values fail

End-to-end fixture

Prove one Nix-authored registry projected to committed JSON and consumed by:

• Nix evaluation
• Rust include_str!/serde
• TypeScript/Effect Schema
• compiled Bun static import

All consumers must reject an unsupported schema version.

Verify by breaking it

For each important guard, deliberately create the failing state, capture nonzero failure, restore, and prove PASS:

1. edit generated bytes by hand
2. change an authority input without regeneration
3. corrupt the embedded fingerprint
4. remove a semantic source from provenance
5. introduce a duplicate/referential-integrity violation
6. make the generated output writable
7. make the generated target set empty
8. bump to an unsupported consumer schema version

Adoption plan

1. Extend and harden projectionArtifact.json() without changing existing callers.
2. Adopt the contract profile in the next Nix -> JSON -> Rust/TS projection, preserving its repo-local artifact schema.
3. Migrate pnpm-install-contract.json.genie.ts and one existing multi-target family as reference adopters.
4. Add a shared devenv task/check facade and consumer recipes.
5. Migrate downstream families opportunistically; delete local hashing/check/write plumbing only after behavior is equivalent and red/green probes pass.

No current downstream correctness fix should block on this extraction. New local implementations should keep authority, schema, and consumers migration-ready.

Tradeoffs

Extend Genie (recommended)

Reuses discovery, deterministic write/check, formatting, validation, and read-only publication. It directly strengthens an existing projection abstraction rather than creating a competing system.

Cost: non-TypeScript authorities need a carefully bounded adapter, and #831's runtime/TUI split should remain compatible.

New standalone Effect library

Could provide excellent typed canonicalization/errors, but by itself does not solve Nix/Rust producers, repository publication, or freshness tasks. Factor pure primitives later only if multiple non-Genie callers justify it.

Nix-only helper

Natural for Nix authorities but fails to cover Rust/TypeScript/remote authorities and would duplicate Genie's publication/check behavior.

Universal schema/codegen tool

Could generate every consumer language, but overgeneralizes extraction, schema semantics, and renderer needs. Existing Weaver, OpenAPI, Notion, Rust schema, and asset pipelines should remain specialized.

Non-goals

• A universal cross-language IDL
• Generating rich Effect Schema semantics from Rust/JSON Schema (tracked separately in #916)
• Replacing Weaver, OpenAPI, Notion introspection, schemars, or asset-specific generators
• A runtime projection service or dynamic registry lookup
• Domain-specific secret/persona/account policy in Genie
• Committing ordinary Nix store outputs or build caches
• One fingerprint shared by targets with different semantic inputs
• Hiding reviewable generated contracts from diffs

Acceptance criteria

• Existing projectionArtifact.json() callers remain supported.
• A documented authority/projection/consumer contract is implemented on top of Genie.
• Embedded JSON provenance and semantic-input fingerprinting are deterministic and validated.
• Multi-target families can use target-specific fingerprints.
• One non-TypeScript authority is supported without duplicating its model in TypeScript.
• Generation/check is atomic, read-only, non-empty, and identical locally and in CI.
• Nix, Rust, TypeScript/Effect, and compiled Bun consumers pass a shared fixture and reject unsupported schemas.
• Every listed red/green stale-artifact probe is demonstrated.
• At least two real adopters replace local plumbing without weakening their existing checks.
• Migration guidance states what is replaced, not only what is removed.

Posted on behalf of @schickling

field	value
agent_name	🔄 co2-tide
agent_session_id	b11aaa26-d085-4ef3-adca-7a64735035c9
agent_tool	Codex CLI
agent_tool_version	0.144.1
agent_runtime	Codex CLI 0.144.1
agent_model	unknown
runtime_profile	/nix/store/9pwivx3z8bh69fjrsziw62ggmipmi35p-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest	/nix/store/y9zyjnc2d5mkgm8p7pv5xd10prcha6z0-agent-skills-corpus/share/agent-skills/manifest.json
worktree	dotfiles/main
machine	mbp2025
tooling_profile	dotfiles@unknown-dirty

Start with packages/@overeng/genie/src/runtime/projection-artifact/mod.ts, then compare its existing API with the proposed contract wrapper and the genie/weaver-registry examples. Build the bounded API prototype and non-TypeScript producer proof, then verify the listed deterministic, fingerprint, validation, freshness, and consumer-rejection cases; done means the chosen contract shape is demonstrated without changing existing callers.