samchon / samchon/compiler-knowledge-graph
[Experiment/index-time] Bind every cell to its intended and executed strict route
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 6
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
Problem
The real-provider and cold-index matrices do not bind every result to both the
strict route they intended to prove and the producer that actually served it.
The measurement can preserve a fallback as useful evidence, but its report and
summary must not turn that fallback into a successful measurement of a
different provider.
The complete index-time run 33377586947 on PR #164 head b26fe40 provides
three independent witnesses:
| Project | Intended owner | Actual result | Current interpretation |
|---|---|---|---|
| Excalidraw | ttscgraph |
static fallback | correctly marked non-semantic, but the report binds provisioned 0.25.0 while the fixture-resolved binary is 0.22.0 |
| Gson | javac-graph |
scip-java |
summary reports a 2.1x strict-provider ratio |
| Redis | clangd-snapshot |
scip-clang |
summary reports a 0.9x strict-provider ratio |
index-time-summary.mjs currently defines “strict provider served” as any
lsp/hybrid servedBy value that does not contain “no strict provider”. A
fallback provider therefore satisfies the predicate even when the primary
owner failed. The report's strict boolean records that strict providers were
enabled, not that the intended owner served.
The Excalidraw artifact separately proves a toolchain-binding gap. Its manifest
records workspace ttscgraph 0.25.0 and its SHA-256, while the pinned fixture
declares ttsc 0.22.0. Project-local resolution wins and the log says that the
executed binary returned a legacy full dump. The report is therefore bound to
provisioned bytes that did not produce the cell.
The real-provider artifacts have a related evidence omission. They record the
provider's declared fact list and observed edge counts, but not the real dump's
15 coverage states or unresolved reason counts. A real producer can therefore
run while the artifact cannot show which families it called complete, partial,
or unsupported.
Consequence surface
- Index-time
report.json, validation, folding, summaries, workflow logs, and
tracked publication inputs. - CLI dump provenance emitted beside discarded large payloads.
- Experiment JSON for every strict provider and the provider-support evidence
that consumes those experiments. - The Excalidraw fixture pin and corpus manifest.
- #72, #73, and #74 acceptance evidence, plus #167's requirement that each
intended strict route either serve or be explicitly adjudicated.
The index-time workflow is a measurement rather than a merge gate. A provider
failure should still produce and upload a report; making the job red would lose
the evidence this issue exists to preserve.
Approach
Give a cell separate structured fields for intent and outcome. Derive the
intended primary owner from the canonical provider registry, not from corpus
names or summary regexes. Preserve the actual provider provenance and producer
version from the completed dump in a bounded machine-readable side channel, so
the report can distinguish the route, tool, and version that served.
Update the Excalidraw source fixture to a protocol-v1 ttsc release and pin the
new fixture commit through the benchmark corpus and question manifest. The
fixture's own project-local tool and the provisioned manifest must describe the
same executable generation.
Add compact coverage and unresolved summaries to real-provider experiment
artifacts. Keep full unresolved sites in the graph dump; the experiment needs
counts by family/state/reason, not another unbounded payload.
Acceptance and verification
- A completed strict-intent cell records the expected primary provider, actual
provider provenance, producer identity/version, and an explicit
served/fallback/static/unknown verdict. - Gson through
scip-javaand Redis throughscip-clangare reported as
primary-provider failures and never receive strict-provider speed ratios. - A same-run fallback remains publishable evidence and the measurement job
remains successful while the summary emits an explicit GitHub warning. - Excalidraw resolves a protocol-v1 project-local
ttscgraph; the actual
provenance agrees with the toolchain evidence and the cell is not static. - Experiment artifacts record all 15 coverage families and unresolved counts
by stable reason for the exact real generation. - Report validation rejects a reversed intent, a mismatched expected owner, and
claimed toolchain/provenance that cannot describe the served producer. - Deterministic positive, fallback, static, timeout, missing-provenance, and
historical-result cases pass without fixture-name branches. pnpm --filter @samchon/graph-benchmark test, focused experiment-contract
tests,pnpm build,pnpm test, andpnpm coveragepass.
Coordination
The next complete index-time matrix must also exercise the provider fixes under
#73 and #74. Do not publish tracked result JSON or chart assets from a partial
matrix, and do not run a paid agent benchmark for this issue.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with index-time-summary.mjs and the index-time report and validation paths described in the issue, then inspect the real-provider experiment artifacts and Excalidraw fixture manifest. Run the focused experiment-contract tests before the full pnpm benchmark suite. Done means intent, actual provenance, toolchain evidence, coverage summaries, and fallback verdicts validate correctly without blocking evidence collection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100