Heuristic signal quality: hidden-link recovery and labelled noise review

Parent: #17

Depends on the evaluation scaffold and canonical correctness suite. Nimbus may be used for mechanics, but final usefulness estimates require pinned real corpora.

Outcome

Measure whether advisory heuristics surface actionable results rather than merely computing their formulas correctly.

Scope

• Frozen development/holdout splits grouped to prevent document or mutation leakage.
• Hidden-link recovery with eligible-link rules, exclusions, and fixed folds.
• Baselines appropriate to each ranking: random, alphabetical, degree, common-neighbour, topology-only trail order, and direct grep.
• Blind multi-reviewer labels for actionable, acceptable/intentional, incorrect/noisy, and uncertain cases.
• Stratified sampling across heuristic kind, score band, corpus, emitted items, and eligible non-emitted controls.
• False-positive taxonomy, adjudication, agreement, and append-only raw labels.
• Development-only threshold tuning followed by one frozen holdout evaluation.

Metrics

Precision@K, Recall@K, MRR, nDCG@K, candidate coverage, actionability rate, noise rate, uncertain rate, agreement, and per-corpus/per-kind confidence intervals.

Constraints

• Existing links are proxy positives, not universal proof that a link should exist.
• Absent links are not negatives.
• Do not tune from Stage A outcomes or inspect holdout labels before freezing thresholds.

Acceptance

• Eligibility, sampling, baselines, K values, labels, and thresholds are frozen before holdout scoring.
• Top-ranked and random/control candidates are reviewed.
• Every advisory heuristic receives a keep/tune/demote/data-only/no-evidence result.
• Under-linked and hops/far-from-root are compared on the same eligible documents.
• Raw labels and generated reports are reproducible and append-only.

Start with the evaluation scaffold and canonical correctness suite, then define the frozen development and holdout splits, eligibility rules, baselines, sampling, labels, and thresholds described here. Done means reproducible append-only raw labels and reports, frozen holdout scoring, and a keep/tune/demote/data-only/no-evidence result for every advisory heuristic.