forrtproject / forrtproject/flora-preprint-notifier

FLoRA screening misses self-replications: preprints emailed about citing themselves

Open
#69 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## Summary

FLoRA screening did not exclude pairs whose replication DOI (`doi_r`) is the preprint itself. As a result, a preprint that *is* a replication study can be flagged eligible and the authors emailed urging them to cite their own paper. Discovered after the email to the authors of `n9kgz_v2` (`10.31234/osf.io/n9kgz_v2`) on 2026-05-07.

## Trace

Preprint `n9kgz_v2` ("Revisiting regulatory fit and its effect on honesty: A replication attempt") cites Achar & Lee (2022), `10.1037/pspa0000306`. FLoRA lists two replications for that original:

1. `10.1177/08902070251410250` — the published EJP version of the same paper
2. `10.31234/osf.io/n9kgz_v2` — this preprint itself

The screening rule in `osf_sync/augmentation/flora_screening.py`

```python
replication_cited = any(d in all_dois for d in replication_dois)
```

builds `all_dois` only from the preprint's *cited references*. Since a preprint never cites its own DOI, neither replication DOI is found, the ref becomes eligible, and the email is sent.

The OSF version-aware deduplication logic in `dynamo/preprints_repo.py:505-559` worked correctly for this preprint (`n9kgz_v1` was excluded as `superseded_by_newer_version` by `n9kgz_v2`). The bug is **not** in OSF version dedup — it's in FLoRA screening missing the self-reference check.

## Audit (production DB, run 2026-05-07)

Across all 59,884 references with FLoRA pairs (covering 2,434 preprints):

| Class | Total affected | Of which only-self replications |
|---|---:|---:|
| Already emailed | 1 | 0 |
| FLoRA-eligible, not yet emailed | 2 | 2 |
| Other | 0 | 0 |

Affected preprints:

- `n9kgz_v2` — **already emailed** (msg `cabe90f90d2b42c8`, treatment arm). The OSF self-pair has been filtered, but the published-version pair `10.1177/08902070251410250` (= same paper, different DOI) still slips through because OSF's `attributes.doi` link is unset. Email already went out.
- `ydgh9_v2` — eligible, not yet emailed; only-self matches. Should be screened ineligible after re-running.
- `x394e_v4` — eligible, not yet emailed; only-self matches (older OSF versions of itself). Should be screened ineligible after re-running.

## Fix landed

`osf_sync/augmentation/flora_screening.py`:

- New helper `_build_self_dois(preprint)` returns the set of DOIs that identify the preprint itself: the linked version-of-record (`doi`), the OSF preprint DOI from `links.preprint_doi`, the unversioned base, and all sibling-version variants (or `_v1..v30` if the record is unversioned).
- `_process_preprint` now fetches the preprint metadata, builds `self_dois`, and drops any FLoRA pair whose `doi_r` matches before persisting `flora_ref_pairs` or computing `replication_cited`. Refs left with no non-self pairs are skipped entirely.

Tests: `tests/test_flora_screening_self_filter.py` (7 cases, all pass).

## Remaining gaps

1. **Unlinked version-of-record DOIs** — when an author publishes the preprint elsewhere but does not register the resulting DOI on OSF, the preprint record's `doi` field stays `None` and `_build_self_dois` cannot include the published DOI. `n9kgz_v2`'s second self-pair (`10.1177/08902070251410250`) is exactly this case. Possible follow-ups: (a) Crossref `relation` lookup from preprint DOI → version-of-record, (b) author-overlap heuristic between preprint authors and `doi_r` author list.
2. **Already-sent email to `n9kgz_v2`** — cannot be unsent. Action items: decide whether to send a corrective note to the recipients, and whether to mark this preprint's trial assignment as protocol-deviated for the analysis.

## Backfill / remediation needed

- [ ] Re-run FLoRA screening for `ydgh9_v2`, `x394e_v4` (and any future preprints flagged) so they're correctly marked ineligible before the next email batch.
- [ ] Decide handling of `n9kgz_v2` (already emailed) per protocol.
- [ ] (Optional) Track unlinked version-of-record self-detection as a separate enhancement issue.

## Reproduce

```bash
python scripts/audit_flora_self_replications.py --out /tmp/flora_self_audit.json
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with osf_sync/augmentation/flora_screening.py and tests/test_flora_screening_self_filter.py, then run the named audit script to inspect the affected records. Done means confirming the self-filter tests and re-screening ydgh9_v2 and x394e_v4; n9kgz_v2 still requires a protocol decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.