Proposal: factory-graph — derived traceability graph rehydrated from .factory/ markdown to eliminate identifier cite drift

# Proposal: `factory-graph` — a derived, queryable traceability graph rehydrated from `.factory/` markdown

## Problem

LLM agents struggle to keep the factory's cross-referenced identifiers (BC-S.SS.NNN, ADR-NNN, VP-NNN, stories, epics, indexes) aligned as changes are made. The drift is not in the source artifacts — it is in **derived data replicated by hand**:

- The four INDEX files (BC-INDEX, VP-INDEX, STORY-INDEX, ARCH-INDEX) whose version numbers are cross-cited in dozens of files and must be bumped + sibling-swept on every artifact touch.
- BC version cells copied into story tables (e.g. "BC-5.42.001 … v1.4").
- Section anchors (`ADR-NNN §Decision N`) and `file.rs::test_fn` citations that go stale when targets move.

Evidence this is the current bottleneck:

- E-19 reached **adversarial pass 29** with a 0/3 clean streak; nearly all recent findings are citation misalignment (e.g. F-P29-001 wrong-crate architecture anchor, F-P28-001 stale VP-096 title across 3 files, F-P29-002/TD-VSDD-091 volatile line pin), not behavior bugs.
- POLICY 5 has been extended six times (META-LEVEL 31→36) trying to fix cite drift with prose rules.
- `validate-index-cite-refresh` exists solely to police the 4-index version-cite ledger, via hand-rolled string scanning.
- Prior structural fixes for the same defect class: ADR-016 (path registry after invented paths), TD-031 (316 line-number citation violations at rollout), the "SHA Drift & Narrative-Staleness" lessons doc (six consecutive recurrences).

Today's alignment machinery is string-level token checks (WASM hooks) plus LLM sweeps (adversary, consistency-validator, spec-steward). **Nothing actually parses the reference graph.**

## Research summary (evaluated alternatives)

The original question was whether a GraphQL/graph database could be checked into git by dehydrating to text and rehydrating at runtime. Candidates evaluated:

| Option | Verdict |
|---|---|
| **Derived-index pattern (recommended)** | Markdown+frontmatter in git IS the database; rehydrate an in-memory graph at session start. No second replica, no sync problem, zero CI infra. |
| **Oxigraph + N-Quads in git** | Best true dehydrate/rehydrate implementation: embedded pure-Rust triple store (Apache-2.0, actively maintained), line-oriented `.nq` files diff/merge cleanly, SPARQL 1.1. Escalation path if purpose-built queries are outgrown. |
| **TerminusDB** | Only candidate with native GraphQL + in-DB git-like clone/branch/diff/merge, but it is a Docker-based SWI-Prolog server (not embeddable) and full-fidelity export is a proprietary binary bundle. Defer unless in-DB branching becomes load-bearing. |
| **Dgraph** | Alive (Istari Digital), native GraphQL, but a server cluster with no git semantics — operational overkill. |
| **Kùzu** | Upstream archived Oct 2025 (Apple acqui-hire); LadybugDB fork is Cypher-only. |
| **SQLite + sqlite-diffable + GraphQL layer** | Viable but adds a redundant markdown→SQL→NDJSON hop. |
| **Dolt** | Git semantics but SQL, binary prolly-tree storage (not git-diffable), no GraphQL. |
| **CozoDB / IndraDB** | AGPL-style copyleft / custom-API-only respectively. |
| **Contentlayer/Gatsby-style TS tools** | Contentlayer unmaintained; Velite/Astro Content Layer viable only in a TS host. |

**Conclusion:** a standalone database dehydrated into git would create a second replica of data that already lives in git — replicas needing sync are precisely the current failure mode. The repo effectively checked its database into git the day it adopted frontmatter; what is missing is the parser and the queries. No GraphQL server is required — agents interact via CLI/hooks; `async-graphql` can be layered on later if wanted.

## Proposed design

New crate: **`factory-graph`** (under `crates/`), consumed by both `factory-dispatcher` hooks and a standalone CLI.

**Rehydration:** parse all of `.factory/` per invocation — frontmatter via a Serde-based frontmatter parser, body references (tables, `ADR-NNN §…` cites, `file::test` anchors) via `pulldown-cmark` — into a `petgraph` directed multigraph. ~2,300 small files should parse sub-second in Rust; if hook latency budgets demand it, add an mtime-hash keyed on-disk cache (a memo, not a database). Reuse `plugins/vsdd-factory/config/artifact-path-registry.yaml` (ADR-016) as the discovery manifest.

**Graph model — typed directed multigraph, NOT a global DAG:**
- Node keys: existing global IDs (POLICY 1 append-only numbering guarantees stability); composite keys for file-scoped sub-artifact IDs, e.g. `S-19.01/AC-003`, `BC-5.42.001/PC-2`.
- Edge type enum: `DependsOn`, `TracesTo`, `Verifies`, `Supersedes`, `Replaces`, `Cites`, `RelatedTo`, …
- Per-edge-type policy table:
- **Must be acyclic:** story/epic `depends_on` (waves = toposort levels; phase-f3's "no cycles" rule becomes deterministic), the spec hierarchy (Brief → CAP → BC → VP → Story → Test), lifecycle chains (`supersedes`/`superseded_by`, `deprecated_by`/`replacement`).
- **Symmetric-consistent:** bidirectional traceability cites (BC §Traceability ↔ story BC-table) are ONE semantic edge materialized in two files — stored once; the two textual appearances become projections, turning "both files must agree" from a consistency rule into a rendering guarantee.
- **Unconstrained:** the process graph (findings → BCs, decisions → findings, policies → decisions) legitimately loops over time; `RelatedTo` is symmetric.
- Treat append-only `last_amended` frontmatter changelog blobs as opaque (do not parse).

**Query CLI (agent-facing, via skill/MCP):** `factory-graph refs `, `orphans`, `stale-cites`, `check-cycles `, `impact ` (transitive closure), `waves`.

**Hooks (dispatcher-facing):** PostToolUse validation on `.factory/` edits — dangling references, version-cell mismatches, dead anchors, per-type cycle violations — replacing string-level checks with graph-level ones.

**Generated indexes:** BC-INDEX / VP-INDEX / STORY-INDEX / ARCH-INDEX become `factory-graph generate-indexes` output (lockfile model: regenerated same-burst, hand-editing is a policy violation). This retires the 4-index version-cite ledger, sibling sweeps, and `validate-index-cite-refresh` wholesale.

## Implementation order

Sequencing principle: **observe → query → advise → enforce → generate → retire.** Each phase proves the graph correct before granting it more authority — a parser with false positives that blocks writes would be worse than the current drift.

1. **Prototype parser (read-only spike).** Parse `.factory/`, build the graph, report: rehydration wall-time + broken-reference counts by category (dangling cites, stale version cells, dead anchors, orphans). *Exit criterion: reproduces known F-* findings from recent adversarial passes without hand-tuning.*
2. **`factory-graph` crate + formalized schema.** Composite ID scheme, edge-type enum, per-type policy table, path-registry-driven discovery. Land the design as an ADR (the edge taxonomy is an architecture decision).
3. **Query CLI for agents** (skill-exposed). First user-visible value, zero risk; adversary/consistency-validator cite tool output instead of prose-scanning.
4. **Advisory hooks (report-only).** Run alongside existing string hooks for several waves; compare findings; measure hook latency; add the cache only if needed. Burn-in catches parser gaps (odd table formats, legacy files) before enforcement.
5. **Blocking enforcement, one check at a time.** Highest-confidence first (dangling refs, lifecycle cycles) → fuzzier last (body-table version cells). Each flip retires the corresponding prose rule and/or string hook; codify each as a POLICY with `enforced_by` set.
6. **Generated indexes (payoff phase).** Most disruptive: many prompts, skills (`register-artifact`, `state-burst`) and policies reference index bumping — budget a prose sweep of `plugins/vsdd-factory/`. Do only after enforcement is stable so the generator provably reads a correct graph.
7. **Downstream + cleanup.** `factory-graph waves` replaces LLM-authored wave math; reimplement `check-input-drift` on the graph; retire redundant META-LEVEL policy extensions; update agent prompts. Optional extras only here: `async-graphql`/MCP surface, or Oxigraph+N-Quads if ad-hoc SPARQL is ever genuinely needed.

**Warnings:** (a) do not start at phase 6 — generating indexes from an unvalidated parser launders parser bugs into canonical artifacts; (b) phases 1–3 are a standalone-value spike; phases 4–6 modify the factory's enforcement machinery while it runs and should flow through the factory's own change management (ADR, stories, policies).

## Migration hazards (from repo analysis)

- Sub-artifact IDs (`AC-NNN`, `PC-N`, `EC-NNN`) are file-scoped, not globally unique → composite keys required.
- Giant append-only nested-prose changelogs inside YAML frontmatter (BC-INDEX `last_amended` is tens of KB) → treat as opaque blobs.
- Hooks + dozens of agent prompts hard-code the layout via `artifact-path-registry.yaml` → conveniently reusable as the loader's manifest.
- Two historical ID renumbering migrations (BC-AUDIT→BC-S.SS.NNN, S-N.M→S-N.MM) have mapping files that should become explicit `ReplacedBy`-style edges so legacy cites resolve.

## Expected wins

- Cite drift caught deterministically at write time instead of at adversarial pass N — directly attacks the defect class that has dominated recent passes.
- The 4-index version-cite ledger ceases to exist as something agents maintain.
- "Both files must agree" bidirectional cites become generated projections of a single edge.
- Wave scheduling, orphan detection, impact analysis, and input-drift checks become cheap deterministic queries instead of LLM prose rules.

The work involves creating a new crate `factory-graph` under `crates/`. Start by examining the `.factory/` directory structure and the `artifact-path-registry.yaml` file to understand the artifact layout. The first step is a read-only prototype parser to build a graph from markdown frontmatter and body references, using libraries like `pulldown-cmark` and `serde`. The exit criterion is reproducing known findings from recent adversarial passes.

Written by the indexing model from the issue text.