camunda / camunda/api-test-generator

chore: promote RDF spike's index-parity check to a regression invariant

Open
#77 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
0
Forks
3
Avg merge
13h 41m
Merged PRs (30d)
23

Description

Context

The RDF spike (#60) graduated with a phase-3 checkpoint that re-materialises three of graphLoader's reverse indexes from SPARQL queries against an oxigraph triple store, and asserts byte-level parity against the live loader output. The script lives on spike/rdf-issue-60 at:

  • docs/spikes/rdf/parity/index-parity.ts
  • docs/spikes/rdf/adapters/build-store.ts
  • docs/spikes/rdf/ontology/{core,camunda}.ttl
  • docs/spikes/rdf/shapes/invariants.shapes.ttl

It is currently a one-shot CLI script that prints PARITY: PASS/FAIL. There is no CI gate enforcing parity — if the loader's output drifts from what the ontology can express, nothing complains.

#66 (Spike #60 follow-ups) lists this as the last remaining backlog item.

Goal

Promote the parity check to a permanent regression invariant under tests/regression/, so that any future change to graphLoader, domain-semantics.json, the ontology, or the adapter that breaks the loader↔store correspondence fails CI.

Scope

This is a sizeable graduation, not a thin wrapper:

  1. Bring spike assets into main:

    • Cherry-pick spike/rdf-issue-60 so docs/spikes/rdf/{adapters,ontology,parity,shapes,...} lands on main.
    • Decision needed: keep under docs/spikes/ (signals "still a spike, just type-checked & gated") or relocate to a non-docs/ location like tests/regression/rdf-parity/{adapters,ontology}/ (signals "permanent test asset"). The latter is more honest about the lifecycle.
  2. Add runtime deps: oxigraph ^0.5.x, n3 ^2.x to the path-analyser workspace (or to a new rdf-parity workspace if we want to keep the SDK boundary clean).

  3. Wrap as vitest: a new tests/regression/rdf-parity.test.ts that:

    • Calls loadGraph(baseDir) and buildStore(baseDir).
    • Computes the three index diffs (bySemanticProducer, domainProducers, providerMap).
    • Strict ontology-rejects policy (changed from spike default): a divergence whose key is 'undefined', '', or 'null' is a failure, not a tolerated artifact. The spike originally surfaced these as warnings on the assumption SHACL would catch them at load time — but SHACL is not enforced anywhere on main, and the loader silently writing domainProducers['undefined'] is itself a defect class we want to guard against. Test must expect(diffs).toEqual([]) after the strict filter.
  4. Class-scoped guard rationale (per AGENTS.md): the test must reject the class of defect "loader and ontology disagree about which operation produces which type/state", not just the three specific reverse indexes the spike happened to query. The three current SPARQL queries cover the BFS-visible surface; if a new BFS-visible reverse index is added to OperationGraph later, this test will not yet cover it. A short comment + checklist in the test header should call that out so reviewers add a fourth SPARQL query when extending the loader.

  5. Pre-push integration: confirm vitest can drive the script under the existing npm test flow without a network or filesystem dependency beyond the bundled spec. The spike script reads path-analyser/operation-dependency-graph.json + path-analyser/domain-semantics.json, both produced by npm run testsuite:generate. The CI step ordering already runs testsuite:generate before npm test, so the test can assume those files exist.

Open design questions

  • Workspace placement of the adapter + ontology + test (see point 1).
  • oxigraph runtime cost: it's a native module (Rust → wasm/native). Confirm CI resolves the right binary on Linux x64 without a pre-build step. If not, either pin to the wasm build or add a CI install step.
  • Performance budget: spike ran the parity check in <2s locally on the bundled spec. Document a soft ceiling (e.g. 10s on CI) and fail loud if it regresses by 5×.
  • Ontology evolution: when a new domain-semantics field lands (e.g. a future capabilities[T].witnesses shape), the adapter and one or more SHACL shapes will need updates. Document this in the test's header comment so it's discoverable.

Acceptance criteria

  • docs/spikes/rdf/ (or relocated equivalent) is on main and type-checks under the relevant workspace tsconfig.json.
  • oxigraph + n3 are declared deps in the chosen workspace and npm ci resolves them on Linux CI.
  • tests/regression/rdf-parity.test.ts runs in npm test and asserts byte-level parity for bySemanticProducer, domainProducers, providerMap.
  • The strict ontology-rejects-fail policy is in place (no undefined/''/null key tolerance).
  • The test header documents the class-scoped guard rationale and the "add a SPARQL query when adding a reverse index" extension contract.
  • CI passes; pre-push checklist (lint + 3× tsc + testsuite:generate + generate:request-validation + npm test) is green locally.

Out of scope

  • Replacing the loader with a SPARQL-backed implementation (the spike's longer-term hypothesis). Parity-as-gate is a stepping stone: it locks in the loader↔ontology correspondence so a future refactor can swap the implementation under it without regressing.
  • Promoting the SHACL shapes (invariants.shapes.ttl) to a load-time gate — separate follow-up.
  • Promoting the other spike queries (minimal-scenario-chain, value-binding-drift) — separate follow-ups.

Related

  • #60 (original spike)
  • #66 (spike follow-ups, this is the last remaining item)
  • spike branch: spike/rdf-issue-60
  • spike commit: 34524b5 chore(spike-rdf): phase 2+3 — adapters + index-parity checkpoint PASS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing docs/spikes/rdf/parity/index-parity.ts, docs/spikes/rdf/adapters/build-store.ts, the ontology and shapes, plus the relevant workspace package and CI configuration. Then trace how the generated graph files are produced before designing tests/regression/rdf-parity.test.ts around the existing vitest flow. Done means the assets and dependencies resolve in CI, strict parity checks cover the three named indexes, and npm test passes without network access.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ci-cd, databases, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.