samchon / samchon/compiler-knowledge-graph

[Experiment/index-time] Bind every cell to its intended and executed strict route

Open
#168 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
6
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Problem

The real-provider and cold-index matrices do not bind every result to both the
strict route they intended to prove and the producer that actually served it.
The measurement can preserve a fallback as useful evidence, but its report and
summary must not turn that fallback into a successful measurement of a
different provider.

The complete index-time run 33377586947 on PR #164 head b26fe40 provides
three independent witnesses:

Project Intended owner Actual result Current interpretation
Excalidraw ttscgraph static fallback correctly marked non-semantic, but the report binds provisioned 0.25.0 while the fixture-resolved binary is 0.22.0
Gson javac-graph scip-java summary reports a 2.1x strict-provider ratio
Redis clangd-snapshot scip-clang summary reports a 0.9x strict-provider ratio

index-time-summary.mjs currently defines “strict provider served” as any
lsp/hybrid servedBy value that does not contain “no strict provider”. A
fallback provider therefore satisfies the predicate even when the primary
owner failed. The report's strict boolean records that strict providers were
enabled, not that the intended owner served.

The Excalidraw artifact separately proves a toolchain-binding gap. Its manifest
records workspace ttscgraph 0.25.0 and its SHA-256, while the pinned fixture
declares ttsc 0.22.0. Project-local resolution wins and the log says that the
executed binary returned a legacy full dump. The report is therefore bound to
provisioned bytes that did not produce the cell.

The real-provider artifacts have a related evidence omission. They record the
provider's declared fact list and observed edge counts, but not the real dump's
15 coverage states or unresolved reason counts. A real producer can therefore
run while the artifact cannot show which families it called complete, partial,
or unsupported.

Consequence surface

  • Index-time report.json, validation, folding, summaries, workflow logs, and
    tracked publication inputs.
  • CLI dump provenance emitted beside discarded large payloads.
  • Experiment JSON for every strict provider and the provider-support evidence
    that consumes those experiments.
  • The Excalidraw fixture pin and corpus manifest.
  • #72, #73, and #74 acceptance evidence, plus #167's requirement that each
    intended strict route either serve or be explicitly adjudicated.

The index-time workflow is a measurement rather than a merge gate. A provider
failure should still produce and upload a report; making the job red would lose
the evidence this issue exists to preserve.

Approach

Give a cell separate structured fields for intent and outcome. Derive the
intended primary owner from the canonical provider registry, not from corpus
names or summary regexes. Preserve the actual provider provenance and producer
version from the completed dump in a bounded machine-readable side channel, so
the report can distinguish the route, tool, and version that served.

Update the Excalidraw source fixture to a protocol-v1 ttsc release and pin the
new fixture commit through the benchmark corpus and question manifest. The
fixture's own project-local tool and the provisioned manifest must describe the
same executable generation.

Add compact coverage and unresolved summaries to real-provider experiment
artifacts. Keep full unresolved sites in the graph dump; the experiment needs
counts by family/state/reason, not another unbounded payload.

Acceptance and verification

  • A completed strict-intent cell records the expected primary provider, actual
    provider provenance, producer identity/version, and an explicit
    served/fallback/static/unknown verdict.
  • Gson through scip-java and Redis through scip-clang are reported as
    primary-provider failures and never receive strict-provider speed ratios.
  • A same-run fallback remains publishable evidence and the measurement job
    remains successful while the summary emits an explicit GitHub warning.
  • Excalidraw resolves a protocol-v1 project-local ttscgraph; the actual
    provenance agrees with the toolchain evidence and the cell is not static.
  • Experiment artifacts record all 15 coverage families and unresolved counts
    by stable reason for the exact real generation.
  • Report validation rejects a reversed intent, a mismatched expected owner, and
    claimed toolchain/provenance that cannot describe the served producer.
  • Deterministic positive, fallback, static, timeout, missing-provenance, and
    historical-result cases pass without fixture-name branches.
  • pnpm --filter @samchon/graph-benchmark test, focused experiment-contract
    tests, pnpm build, pnpm test, and pnpm coverage pass.

Coordination

The next complete index-time matrix must also exercise the provider fixes under
#73 and #74. Do not publish tracked result JSON or chart assets from a partial
matrix, and do not run a paid agent benchmark for this issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with index-time-summary.mjs and the index-time report and validation paths described in the issue, then inspect the real-provider experiment artifacts and Excalidraw fixture manifest. Run the focused experiment-contract tests before the full pnpm benchmark suite. Done means intent, actual provenance, toolchain evidence, coverage summaries, and fallback verdicts validate correctly without blocking evidence collection.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.