forrtproject / forrtproject/flora-preprint-notifier

Deviation from registration: validation runs after randomisation

Open
#70 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## Deviation from registration: validation runs after randomisation

**What happens**
Preprints are randomised at the *potential match* stage. The inbox/citation-distance
check that confirms whether a match is *real* runs afterwards. So a few units are
randomised and only then turn out not to be real matches. This reverses the
registered order (confirm match → randomise).

**Decision**
We are not changing the randomisation partway through the trial. Instead, units that
validation shows are **not real matches** (every eligible reference rejected) are
dropped from the trial, from **both** arms, blind to outcomes.

**Emailing**
The notification step skips rejected matches, so a non-real match normally never
triggers an email. A small number were emailed *before* their validation was
processed, because a separate gate bug let unvalidated matches through (fixed in #71).
Those units are dropped from the analysis like any other non-real match.

**Why this doesn't bias results**
- "Not a real match" is fixed by the validation step, applied identically in both arms
and independent of outcomes.
- Randomising then dropping these units leaves the same analysed population as dropping
them before randomisation, so there is no estimand shift and no bias to the
treatment–control contrast.
- The only cost is a slightly less even treatment/control split within preprint-server
strata (the server is the only stratifier).

**Units dropped (as of this record)**
5 randomised units, both arms — treatment: `3aksc_v1`, `txhgr_v4` (both emailed before
validation); control: `yc6wn_v1`, `c7vnu_v1`, `jcnse_v1`. Marked `excluded` with reason
`non_real_match_post_validation`.

This issue records the deviation and decision before any outcome analysis.

Contributor guide

No contributing guide indexed for this repository

Research direction

This issue records an experimental deviation and decision rather than naming an implementation change. No files, tests, or entry points are mentioned; read the issue alongside the validation and notification work referenced in #71. Done is documenting or verifying the recorded exclusions, not implementing a specified task.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.