forrtproject / forrtproject/flora-preprint-notifier
FLoRA screening misses self-replications: preprints emailed about citing themselves
- Dominant language
- Python
- Stars
- 6
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
FLoRA screening did not exclude pairs whose replication DOI (`doi_r`) is the preprint itself. As a result, a preprint that *is* a replication study can be flagged eligible and the authors emailed urging them to cite their own paper. Discovered after the email to the authors of `n9kgz_v2` (`10.31234/osf.io/n9kgz_v2`) on 2026-05-07.
## Trace
Preprint `n9kgz_v2` ("Revisiting regulatory fit and its effect on honesty: A replication attempt") cites Achar & Lee (2022), `10.1037/pspa0000306`. FLoRA lists two replications for that original:
1. `10.1177/08902070251410250` — the published EJP version of the same paper
2. `10.31234/osf.io/n9kgz_v2` — this preprint itself
The screening rule in `osf_sync/augmentation/flora_screening.py`
```python
replication_cited = any(d in all_dois for d in replication_dois)
```
builds `all_dois` only from the preprint's *cited references*. Since a preprint never cites its own DOI, neither replication DOI is found, the ref becomes eligible, and the email is sent.
The OSF version-aware deduplication logic in `dynamo/preprints_repo.py:505-559` worked correctly for this preprint (`n9kgz_v1` was excluded as `superseded_by_newer_version` by `n9kgz_v2`). The bug is **not** in OSF version dedup — it's in FLoRA screening missing the self-reference check.
## Audit (production DB, run 2026-05-07)
Across all 59,884 references with FLoRA pairs (covering 2,434 preprints):
| Class | Total affected | Of which only-self replications |
|---|---:|---:|
| Already emailed | 1 | 0 |
| FLoRA-eligible, not yet emailed | 2 | 2 |
| Other | 0 | 0 |
Affected preprints:
- `n9kgz_v2` — **already emailed** (msg `cabe90f90d2b42c8`, treatment arm). The OSF self-pair has been filtered, but the published-version pair `10.1177/08902070251410250` (= same paper, different DOI) still slips through because OSF's `attributes.doi` link is unset. Email already went out.
- `ydgh9_v2` — eligible, not yet emailed; only-self matches. Should be screened ineligible after re-running.
- `x394e_v4` — eligible, not yet emailed; only-self matches (older OSF versions of itself). Should be screened ineligible after re-running.
## Fix landed
`osf_sync/augmentation/flora_screening.py`:
- New helper `_build_self_dois(preprint)` returns the set of DOIs that identify the preprint itself: the linked version-of-record (`doi`), the OSF preprint DOI from `links.preprint_doi`, the unversioned base, and all sibling-version variants (or `_v1..v30` if the record is unversioned).
- `_process_preprint` now fetches the preprint metadata, builds `self_dois`, and drops any FLoRA pair whose `doi_r` matches before persisting `flora_ref_pairs` or computing `replication_cited`. Refs left with no non-self pairs are skipped entirely.
Tests: `tests/test_flora_screening_self_filter.py` (7 cases, all pass).
## Remaining gaps
1. **Unlinked version-of-record DOIs** — when an author publishes the preprint elsewhere but does not register the resulting DOI on OSF, the preprint record's `doi` field stays `None` and `_build_self_dois` cannot include the published DOI. `n9kgz_v2`'s second self-pair (`10.1177/08902070251410250`) is exactly this case. Possible follow-ups: (a) Crossref `relation` lookup from preprint DOI → version-of-record, (b) author-overlap heuristic between preprint authors and `doi_r` author list.
2. **Already-sent email to `n9kgz_v2`** — cannot be unsent. Action items: decide whether to send a corrective note to the recipients, and whether to mark this preprint's trial assignment as protocol-deviated for the analysis.
## Backfill / remediation needed
- [ ] Re-run FLoRA screening for `ydgh9_v2`, `x394e_v4` (and any future preprints flagged) so they're correctly marked ineligible before the next email batch.
- [ ] Decide handling of `n9kgz_v2` (already emailed) per protocol.
- [ ] (Optional) Track unlinked version-of-record self-detection as a separate enhancement issue.
## Reproduce
```bash
python scripts/audit_flora_self_replications.py --out /tmp/flora_self_audit.json
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with osf_sync/augmentation/flora_screening.py and tests/test_flora_screening_self_filter.py, then run the named audit script to inspect the affected records. Done means confirming the self-filter tests and re-screening ydgh9_v2 and x394e_v4; n9kgz_v2 still requires a protocol decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100