Explore first-class Rust ↔ rich Effect Schema interoperability and generation

Problem

We now have a second concrete Rust ↔ Effect boundary (dotfiles task-system lifecycle work in dotfiles #1105) that wants to reuse and refine the rollout contract pattern.

The existing proven pipeline is:

Rust serde + schemars (wire source of truth)
  → deterministic committed JSON Schema
  → generated TypeScript types
  → shared positive/negative samples
  → Rust jsonschema validation
  → hand-authored rich Effect Schema decoders
  → bidirectional conformance + shape-identical round trips

This is substantially better than independent untested wire models, but the Effect runtime schema is still a semantic hand-mirror. That duplication becomes risky as more infra-level Rust services expose protocols to Effect applications.

A types-only or AJV-only endpoint is explicitly not the desired end state. Rich Effect Schemas are a hard requirement at the TypeScript edge: annotations, transformations, branded/refined domain values, structured parse errors, and composition with Effect workflows must remain first-class.

Existing prior art

The dotfiles rollout Rust-port prototype proved:

• 10/10 Rust → JSON Schema → production Effect decoder conformance cases
• deterministic schema and TypeScript generation
• tagged/literal-union compatibility
• shared runtime validation against the committed schema
• positive and negative refinement controls
• cross-language shape-identical round trips

It also found concrete impedance mismatches:

• Serde deny_unknown_fields / additionalProperties: false can be stricter than Effect's accept-and-strip excess-property behavior.
• Schemars optionals may emit nullable values where the Effect contract means absent-or-present.
• Opaque JSON values can become over-constrained by generated TypeScript.
• Both schema and TypeScript generators must be pinned and freshness-gated.
• Generated TypeScript types do not provide trust-boundary runtime decoding.
• JSON Schema captures wire structure but not all Effect-domain transformations, annotations, refinements, or structured errors.

Relevant downstream design/evidence:

• context/nix-convergence/03-rollout/.decisions/09-implementation-language-and-portability.md
• context/nix-convergence/03-rollout/.experiments/rust-port-prototype-validation.md
• dotfiles #1105

No existing effect-utils issue found via issue/code search covers Rust/serde/schemars ↔ rich Effect Schema interoperability.

Requirements

1. Preserve one explicit authority for the wire contract, or define a mechanically checkable authority split.
2. Preserve rich, native Effect Schemas—not merely generated TypeScript types or AJV validators.
3. Support Effect annotations, transformations, brands/refinements, structured errors, and ergonomic composition.
4. Make any semantics that cannot cross JSON Schema explicit and freshness/conformance-gated.
5. Generate deterministic, reviewable artifacts with pinned tool versions.
6. Validate both implementations against shared positive and negative vectors.
7. Detect semantic drift, including excess-property, optional/null, discriminant, numeric/string refinement, and transform behavior.
8. Keep the reusable core protocol-agnostic; rollout and task-system should be adopters, not special cases.
9. Avoid forcing domain semantics into Rust wire/storage types when they belong at the Effect application edge.
10. Provide an incremental adoption path from the current rollout pipeline.

Options to investigate

A. Generate Effect Schema from JSON Schema

Generate native Effect Schema source from the Schemars-produced JSON Schema, then layer domain transforms/annotations through explicit extension points.

Upside: one structural wire authority; minimal hand duplication.

Risk: JSON Schema is lossy for Effect semantics. Extension/overlay design must be principled, composable, and freshness-gated; generated schemas must remain idiomatic and debuggable.

B. Restricted shared IDL with Rust and Effect generators

Define a deliberately small contract IDL that generates Rust serde/schemars types and native Effect Schemas, with escape hatches for edge-local domain lifting.

Upside: can model cross-language semantics intentionally rather than inherit JSON Schema's limits.

Risk: creates and maintains a new language/toolchain; risks becoming a lowest-common-denominator schema system or duplicating mature JSON Schema machinery.

C. Rust annotations carrying Effect-generation metadata

Extend Rust types/fields with namespaced metadata (for annotations, brands, refinements, error identifiers, optional/excess semantics) preserved through JSON Schema and consumed by an Effect Schema generator.

Upside: evolves the proven serde/schemars pipeline; reviewable metadata remains near the wire declaration.

Risk: TypeScript-specific semantics leak into Rust; arbitrary transforms cannot safely be serialized as metadata; proc-macro/Schemars extension compatibility needs proof.

D. Hand-authored rich Effect Schemas plus a reusable semantic conformance harness

Retain the current dual executable schemas, but extract first-class tooling that derives/shared-generates vector suites, checks structural and semantic compatibility, performs round trips, mutation-tests boundaries, and freshness-gates artifacts.

Upside: richest Effect ergonomics now; lowest generator-semantic risk; honest about two models.

Risk: conformance testing reduces but does not eliminate duplicated-authority drift; coverage quality becomes load-bearing.

E. Hybrid structural generation + explicit Effect domain overlays

Generate the lossless structural/wire Effect Schema from JSON Schema, then require typed, named overlays for annotations, transformations, refinements, and structured domain errors. The harness proves both the generated base and the lifted domain contract.

Upside: likely best balance: removes structural duplication while preserving rich Effect semantics.

Risk: requires a clean composition API and proof that overlays cannot silently weaken or contradict the wire schema.

Proposed exploration

Prototype at least A, D, and E against the same non-trivial rollout subset and the small task-system lifecycle RPC. Include:

• tagged request/response/event unions
• absent-but-not-null fields
• open-wire / accept-and-strip excess semantics
• opaque JSON
• branded IDs and constrained strings/numbers
• a normalization transform
• structured parse-error annotations
• schema evolution with additive fields and unknown variants
• deterministic regeneration and stale-artifact failure

Compare:

• semantic fidelity
• generated-code readability
• authoring ergonomics on both sides
• quality/locality of errors
• compatibility/evolution behavior
• build/toolchain cost
• amount of handwritten duplication
• ability to compose the result as normal Effect Schema

Acceptance criteria

• A written authority model distinguishes wire shape from Effect-domain semantics.
• At least two real downstream contracts pass shared positive, negative, round-trip, and evolution suites in both Rust and Effect.
• Rich Effect Schema features remain demonstrated, not deferred: annotations, at least one transform, at least one refinement/brand, and structured errors.
• A mutation test proves drift is caught for every known impedance mismatch above.
• Generated artifacts are deterministic and freshness-gated in Nix/CI.
• The selected approach has a reusable effect-utils package/CLI API and migration guide.
• The rollout consumer can adopt it without weakening its current runtime validation.
• dotfiles #1105 can use it for the lifecycle RPC without copying the rollout harness.
• Cleanup removes superseded per-consumer generators/conformance scripts after both adopters migrate.

Non-goals

• Replacing Effect Schema with generated TypeScript types.
• Treating AJV validation alone as the Effect application model.
• Encoding arbitrary Effect transformations as opaque executable strings in Rust metadata.
• Designing a universal cross-language schema language before experiments justify it.

Posted on behalf of @schickling

field	value
agent_name	🗡️ co2-falcon
agent_session_id	0ce4fdf3-a6e4-402a-a019-4d82ad694dde
agent_tool	Codex CLI
agent_tool_version	0.144.1
agent_runtime	Codex CLI 0.144.1
agent_model	unknown
runtime_profile	/nix/store/qq3avrif77r90ypqbdv3hgd3gvwj5s32-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest	/nix/store/pjlb3cwf453ghzmc8jj4v8h29sw6p4dg-agent-skills-corpus/share/agent-skills/manifest.json
worktree	dotfiles/schickling/2026-06-21-buck2
machine	dev3
tooling_profile	dotfiles@4b8e1c5

Start with context/nix-convergence/03-rollout/.decisions/09-implementation-language-and-portability.md and context/nix-convergence/03-rollout/.experiments/rust-port-prototype-validation.md, then inspect dotfiles #1105. Prototype options A, D, and E against the rollout subset and task-system lifecycle RPC. Done means a documented authority model, shared positive/negative/round-trip/evolution suites, deterministic freshness-gated artifacts, mutation coverage for known mismatches, and a reusable migration path.