Luminous-Dynamics / Luminous-Dynamics/symthaea

research-result: forbid positive confirmatory claims from being supported only by Missing/NotComputed outcomes

Open
#1,982 2 comments 0 reactions 1 assignee View on GitHub

@Tristan-Stoltz-ERC is already working on this.

Since Sep 12, 2026.

enhancement
Dominant language
Rust
Stars
9
Forks
1
Avg merge
15m
Merged PRs (30d)
8

Description

Context

Follow-up to PR #159 (symthaea-research-result) and generic research-integrity prerequisite for #1905 / #1898.

The current result layer correctly requires every preregistered primary metric to appear as an explicit outcome, including Missing { reason } and NotComputed { reason }. That is good.

However, the current claim validation checks only that a referenced metric id exists in the result manifest. It does not require the referenced metric to contain an observation capable of supporting the claim disposition.

Therefore a structurally valid manifest can currently express the equivalent of:

metric M = Missing("sensor failed")
claim C = ConsistentWithHypothesis
claim C references M
claim C interpretation = Confirmatory

The metric did not disappear, but the positive claim is still unsupported by the recorded observation state.

Core invariant

metric present
!=
metric observed
!=
metric supports positive/negative claim

A Missing or NotComputed primary result must never, by itself, support a confirmatory ConsistentWithHypothesis, InconsistentWithHypothesis, or NullResult claim.

Proposed evidence-state semantics

Classify MetricOutcome into an explicit support state, conceptually:

ObservedNumeric
ObservedBoolean
ObservedCategorical
Missing
NotComputed

and define conservative claim compatibility.

At minimum:

  • ConsistentWithHypothesis requires at least one referenced observed metric or an explicitly typed observed/result artifact allowed by the frozen analysis plan;
  • InconsistentWithHypothesis likewise requires observed evidence;
  • NullResult requires observed evidence sufficient to apply the preregistered null/no-effect decision rule; absence of measurement is not a null result;
  • Inconclusive / NotEvaluated may be supported by Missing / NotComputed as appropriate;
  • DescriptiveOnly should not be promoted into confirmatory hypothesis support merely by citing an analysis artifact.

Do not hard-code domain statistics into this generic crate. The generic layer should only enforce the distinction between observed evidence and absent/uncomputed evidence; domain/analysis code decides whether an observed value meets a preregistered criterion.

Primary metric completeness vs evidential adequacy

Preserve the current good rule:

all primary metrics must appear

but add a separate rule:

claim evidence must be semantically adequate for its disposition

These should remain distinct.

A campaign may therefore be procedurally confirmatory while yielding an inconclusive claim because a primary observation is missing. Confirmatory describes protocol lineage/analysis status; it must not imply that the scientific hypothesis received positive support.

Claim evidence projection

For each ResultClaim, derive a small evidence summary such as:

observed_metric_ids
missing_metric_ids
not_computed_metric_ids
observed_artifact_ids

Then validate disposition compatibility.

If artifact-only confirmatory hypothesis claims remain supported, require the artifact kind/role to be explicit and bound by the frozen analysis plan rather than treating any arbitrary digested artifact as evidentially interchangeable with an observed metric.

Adversarial regressions

At minimum:

  1. ConsistentWithHypothesis citing only Missing metric -> reject;
  2. ConsistentWithHypothesis citing only NotComputed metric -> reject;
  3. InconsistentWithHypothesis citing only missing/uncomputed metric -> reject;
  4. NullResult citing only missing/uncomputed metric -> reject;
  5. Inconclusive citing an explicit missing primary metric -> allow;
  6. NotEvaluated citing explicit not-computed metric -> allow;
  7. observed numeric metric may support a claim subject to existing protocol/reference checks;
  8. one observed + one missing metric remains explicit; missing evidence is not silently ignored in reporting;
  9. arbitrary Analysis artifact alone cannot launder a missing primary metric into a positive confirmatory claim unless an explicit frozen artifact-evidence rule allows it;
  10. imported/deserialized claim/outcome combinations receive the same semantic validation as constructor-created records.

Relationship to #1946

#1946 closes chronology/completeness/current-head integrity for amendments/deviations and recursively validates imported research evidence.

This issue is complementary: even a complete, correctly ordered result lineage can still make an evidentially invalid scientific claim if Missing is treated as though it were an observation.

Acceptance gate

  • metric presence and metric observation are separate concepts;
  • positive/negative/null hypothesis dispositions cannot be supported only by missing/uncomputed outcomes;
  • missing/not-computed primary metrics remain first-class reported results;
  • procedural confirmatory status does not imply positive hypothesis support;
  • artifact-only hypothesis claims have an explicit conservative rule;
  • claim compatibility is recursively validated on imported manifests;
  • exact-head tests land in #1940 qualification scope.

Non-claims

This does not decide whether an observed effect is scientifically meaningful, statistically significant, causal, or generalizable. It prevents absence of observation from being represented as positive/negative/null evidence.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.