stacklok / stacklok/matlatl

Heuristic correctness: canonical fixtures and independent oracles

Open
#34 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Parent: #17

Depends on the evaluation scaffold issue.

Outcome

Validate the mathematical and resolver correctness of matlatl's shipped mechanisms without using matlatl output as its own oracle.

Scope

  • Canonical empty, singleton, path, star, cycle, SCC, DAG, disconnected, multi-root, bridge, articulation, and symmetric-tie graph fixtures.
  • Resolver fixtures for relative, root-absolute, directory, fragment, alias/wikilink, ambiguous, external, non-note, stale, malformed, and root-escape references.
  • Hand-enumerated or independently calculated expectations for reachability, bow-tie classes, hops, HITS, PageRank, navigability scalars, betweenness, articulation points, bridges, trails, backlinks, scent, knowledge gaps, and suggested links.
  • Seeded reversible graph and reference mutations with expected deltas.
  • Byte-stability and shuffled-input checks.

Constraints

  • Binding definitions are the ADRs listed in docs/research/heuristic-evaluation.md.
  • Any unexplained correctness mismatch blocks usefulness or agent-outcome claims for that mechanism.
  • Floating-point comparison and accumulation order must be explicitly pinned.

Acceptance

  • Every shipped mechanism has at least one independent correctness oracle.
  • Threshold boundaries, exemptions, caps, tie ordering, and indeterminate cases are covered.
  • Invalid universal monotonic assumptions are not encoded as tests.
  • Repeated and shuffled-input runs are byte-stable.
  • The complete offline correctness suite runs through task eval:oracle.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the binding definitions in docs/research/heuristic-evaluation.md and review the evaluation scaffold dependency from issue #17. Build the canonical graph and resolver fixtures with independent expectations, then run task eval:oracle. Done means every shipped mechanism has an oracle, boundary and indeterminate cases are covered, and repeated or shuffled-input runs are byte-stable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.