forrtproject / forrtproject/flora-preprint-notifier
Deviation from registration: validation runs after randomisation
- Dominant language
- Python
- Stars
- 6
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
## Deviation from registration: validation runs after randomisation
**What happens**
Preprints are randomised at the *potential match* stage. The inbox/citation-distance
check that confirms whether a match is *real* runs afterwards. So a few units are
randomised and only then turn out not to be real matches. This reverses the
registered order (confirm match → randomise).
**Decision**
We are not changing the randomisation partway through the trial. Instead, units that
validation shows are **not real matches** (every eligible reference rejected) are
dropped from the trial, from **both** arms, blind to outcomes.
**Emailing**
The notification step skips rejected matches, so a non-real match normally never
triggers an email. A small number were emailed *before* their validation was
processed, because a separate gate bug let unvalidated matches through (fixed in #71).
Those units are dropped from the analysis like any other non-real match.
**Why this doesn't bias results**
- "Not a real match" is fixed by the validation step, applied identically in both arms
and independent of outcomes.
- Randomising then dropping these units leaves the same analysed population as dropping
them before randomisation, so there is no estimand shift and no bias to the
treatment–control contrast.
- The only cost is a slightly less even treatment/control split within preprint-server
strata (the server is the only stratifier).
**Units dropped (as of this record)**
5 randomised units, both arms — treatment: `3aksc_v1`, `txhgr_v4` (both emailed before
validation); control: `yc6wn_v1`, `c7vnu_v1`, `jcnse_v1`. Marked `excluded` with reason
`non_real_match_post_validation`.
This issue records the deviation and decision before any outcome analysis.
Contributor guide
No contributing guide indexed for this repository
Research direction
This issue records an experimental deviation and decision rather than naming an implementation change. No files, tests, or entry points are mentioned; read the issue alongside the validation and notification work referenced in #71. Done is documenting or verifying the recorded exclusions, not implementing a specified task.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100