Intervention pairs can compare different outcome targets under the same instance ID

Summary

The intervention metric estimates a causal effect by comparing Y between control and treatment rows that share an instance ID. That requires Y to represent the same target behavior in both arms.

The scaffold currently loads each row's answer independently and does not verify that paired control/treatment rows for the same intervention instance use the same reference target. If a malformed or edited dataset gives the two arms different answer values, both rows are graded successfully but P(Y=1|X=1) - P(Y=1|X=0) is no longer comparing the same outcome.

This can produce a plausible-looking treatment effect and monitorability score from an invalid causal pairing instead of failing at the dataset boundary.

Expected behavior

Within each intervention dataset and instance ID, the paired rows should agree on the reference answer used to define Y. A mismatch should fail before sampling or metric computation.

Suggested fix

Validate intervention outcome targets after applying any dataset row cap and before building EvalItems. Add regression coverage for a valid pair and for a control/treatment pair with different reference answers.

Locate the intervention dataset loading path and the code that applies row caps and builds EvalItems. Add validation after the cap and before EvalItem construction, then add regression coverage for matching control/treatment answers and for a mismatched pair that fails before sampling or metric computation.