SYM-EVAL-CORE-004: preregister experiment plans, seeds, metrics, and stopping rules

Purpose

Add a versioned pre-run experiment plan artifact to the Symtropy/Symthaea evaluation program so valid blinding cannot be paired with post-hoc seed selection, metric switching, exclusion changes, or stopping-rule drift.

This is a scientific-integrity layer, distinct from cryptographic/runtime integrity:

CORE-001  ordering + hidden-assignment commitment
CORE-002  durable receipt/recovery
CORE-003  capability/oracle isolation
CORE-004  precommitted scientific analysis plan

A technically pristine run can still be scientifically weak if the analysis plan changes after outcomes are visible.

Integration gate

CORE-004 may be designed immediately, but executable integration should reuse qualified CORE-001 artifact commitments rather than introduce a second plan-digest mechanism.

The experiment-plan digest should be included in a pre-run ContractCommitted event before the blinding commitment and before any A/B sensor or producer output.

ExperimentPlanV1

Freeze a closed/versioned plan, conceptually:

ExperimentPlanV1 {
    plan_schema_id,
    plan_schema_version,
    experiment_family,
    scientific_question,
    claim_scope,
    scenario_contract,
    factor_matrix,
    frozen_seed_manifest,
    primary_metrics,
    secondary_metrics,
    negative_controls,
    exclusion_policy,
    retry_policy,
    stopping_rule,
    calibration_policy,
    aggregation_policy,
    hardware_runtime_class,
    producer_identities,
    evaluator_identities,
    blind_stage_profile,
}

Exact names may differ, but the distinctions below are required.

Scientific question / claim scope

Every plan should say what exact proposition is being tested and what conclusions are out of scope.

Examples:

• raw tracker local-ID continuity under bounded occlusion;
• integrated belief freshness/revision under hidden removal;
• calibrated persistence probability under a frozen scenario population;
• causal intervention sensitivity for one scenario family.

Do not register vague claims such as “vision works” or “object permanence solved.”

Frozen seed manifest

Commit the complete seed set or a deterministic seed-generation rule before execution.

The seed contract should bind:

• generator/schema identity;
• scenario generator exact version/head;
• explicit seeds or deterministic derivation inputs;
• number of planned replicates;
• stratification/factor-cell assignment;
• any reserved diagnostic-only seeds.

After any outcome is visible:

• do not drop inconvenient seeds;
• do not add replacement seeds without an explicit new plan/revision;
• do not relabel a diagnostic seed as confirmatory.

A failed infrastructure run may be retried under the frozen retry policy; a scientifically poor outcome is not an infrastructure failure.

Primary vs secondary metrics

Separate metrics before execution.

Primary

These support the main planned inference and should have frozen definitions/thresholds.

Secondary / diagnostic

These may explain failure modes but do not silently replace a failed primary metric.

Exploratory

Post-hoc analyses are allowed if labeled exploratory and bound to a later analysis artifact. They must not be rewritten as preregistered findings.

Metric identity

Each metric definition should bind:

• semantic name;
• schema/version;
• exact eligible producer representation;
• units/coordinate frame;
• aggregation function;
• handling of ambiguous/unavailable samples;
• confidence/calibration eligibility where applicable;
• scorer exact implementation identity.

For example, do not switch from track-ID continuity to entity identity, or from patch-grid distance to box IoU, because one produces a more favorable number.

Negative controls

Pre-register negative controls alongside positive conditions.

For the visual program this may include:

• background-only scenes;
• impossible teleport re-entry;
• shuffled target association;
• hidden duplicate replacement;
• sensor blackout vs world removal;
• random-gaze vs epistemic-gaze controls later;
• shuffled/lesioned representation probes.

A positive result without its planned negative controls should be classified incomplete, not silently promoted.

Exclusion policy

Freeze what can exclude a run/sample before execution, e.g.:

• corrupt sensor artifact;
• digest mismatch;
• runner crash before producer commitment;
• explicit renderer contract violation;
• impossible configuration/schema mismatch.

Do not allow exclusion simply because:

• confidence was low;
• target reacquisition failed;
• a distractor caused a switch;
• a seed produced an unusually hard scene;
• a primary metric worsened.

Every exclusion should appear in the run/aggregate evidence with its exact rule ID.

Retry policy

Distinguish:

infrastructure retry

from:

new scientific observation

Pre-register:

• which failure classes permit retry;
• maximum retry count;
• whether retry reuses the same seed;
• whether a retry receives a new experiment/run ID;
• how pre-reveal vs post-reveal failures differ.

After reveal, a rerun is a new blinded experiment with a new nonce/commitment, not a continuation of the original blind run.

Stopping rule

Freeze the planned sample size or adaptive stopping rule.

Do not stop early because the current result looks favorable unless the stopping rule was precommitted.

If sequential analysis is desired, freeze the exact boundaries/error-spending rule before execution.

Calibration policy

If calibrated probabilities are evaluated, pre-register:

• calibration identity/version;
• calibration population;
• evaluation population;
• binning policy for ECE if used;
• Brier/log-score definition;
• abstention/selective-risk thresholds;
• policy for out-of-distribution scenario strata.

Do not fit or retune calibration on the blinded evaluation outcomes and then score the same outcomes as held-out evidence.

Aggregation policy

Before runs execute, freeze which strata may be pooled.

Examples requiring separate strata unless explicitly justified:

• different renderer/GPU determinism classes;
• different Symthaea heads/configs;
• different camera contracts;
• different producer schemas;
• different calibration identities;
• different scenario-generator versions.

The aggregate should reference immutable CORE-001/002 run heads and exact plan identity.

Hardware / runtime class

Bind the hardware/runtime class relevant to latency/backpressure claims.

Accuracy-only analyses may have broader hardware equivalence, but real-time claims must not silently pool materially different accelerators/CPU/GPU configurations.

Do not invent a real-time PASS threshold after seeing measured throughput.

Plan revisions

A plan is immutable once its first run begins.

Before execution, revisions create a new plan revision with explicit predecessor/reason.

After any result is visible, material changes create a new experiment plan/family revision and future runs only. Historical runs retain the plan under which they were executed.

Do not mutate the original plan artifact in place.

Exploratory follow-up

Exploration is allowed and valuable.

The receipt/report should simply distinguish:

• preregistered confirmatory result;
• preregistered diagnostic result;
• post-hoc exploratory analysis;
• follow-up hypothesis generated for a future plan.

This preserves discovery without laundering hindsight into prediction.

Plan commitment order

Recommended CORE-001 order:

Genesis
  ↓
ExperimentPlanV1 committed
  ↓
scenario/camera/capability contracts committed
  ↓
blinding commitment + precommitted stage profile
  ↓
A/B execution
  ↓
reveal
  ↓
scoring under the precommitted plan

The scorer should reject a confirmatory receipt whose plan commitment is not an ancestor of the first blind output.

Report schema

Final performance receipts should carry:

• plan digest/identity;
• confirmatory vs exploratory classification;
• completed planned sample count;
• excluded sample count + rule IDs;
• retries + reasons;
• primary metrics;
• planned secondary diagnostics;
• negative-control results;
• deviations from plan, if any;
• immutable run chain heads used in aggregation.

Any deviation should remain visible rather than being edited out of the final narrative.

Negative controls for the planning system

Qualification should prove at minimum:

1. first blind output is rejected if no plan is committed for a confirmatory profile;
2. plan digest changes when seed set changes;
3. plan digest changes when primary metric changes;
4. post-start plan mutation is rejected;
5. excluded run requires a registered exclusion rule;
6. unregistered seed cannot silently enter the confirmatory aggregate;
7. planned negative controls cannot be dropped without deviation evidence;
8. post-hoc metric is labeled exploratory;
9. calibration identity/population substitution is detected;
10. incompatible strata cannot pool under a strict aggregation policy;
11. early stopping outside the frozen rule is classified as deviation/incomplete;
12. historical run remains bound to its original plan revision.

Relationship to SYM-EVAL-001D

CORE-004 should become the scientific preregistration layer for the factorial visual stress matrix.

The factor matrix, seed set, primary metrics, negative controls, exclusion rules, and aggregation strata should be committed before large-scale execution.

This is especially important because a rich factorial crucible produces many possible slices; preregistration prevents selecting only the slices that flatter the system.

Nonclaims

CORE-004 does not itself prove statistical validity, adequate power, generalization, causal identification, object permanence, visual reasoning, or model correctness. It establishes only that the declared scientific plan was frozen before outcomes and that deviations/exploratory analyses remain distinguishable.

Start by reading the CORE-001 commitment flow and the SYM-EVAL-001D factorial visual stress matrix, then locate where the pre-run ContractCommitted event and CORE-001/002 run heads are represented. Done means the listed qualification checks cover plan identity, revisions, seeds, metrics, exclusions, retries, stopping, strata, and exploratory classification without mutating historical plans.