forrtproject / forrtproject/flora-extractor
doi_r is never verified, and ~2/3 of the pool's low-id works carry another paper's DOI
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
## Summary
The pipeline verifies `doi_o` against the metadata that DOI actually points to
(`_verify_row` → CrossRef → OpenAlex) and never asks the same question of `doi_r`,
because nothing suggested the pool's own DOIs could be wrong. They can. OpenAlex holds
a class of records whose title, authors and landing page belong to one paper and whose
DOI and journal belong to another:
```
W8856083 doi 10.1111/j.1432-1033.1996.0774h.x
title Legal problems of information filtering techniques
landing dblp.uni-trier.de/db/conf/knowright/knowright95.html#Schweighofer95
journal European Journal of Biochemistry
```
These are not blank or dummy identifiers. Each DOI is a real DOI belonging to a real,
unrelated article.
## How big the class is, and where it lives
Found while deduplicating same-DOI works for #193. Of the pool's 16,374 DOIs held under
more than one work id, 13,172 hold records whose titles disagree — and **12,165 of those
pair a work id under 10^8 with one over 10^9**. The defect is confined to low (early
ingested) OpenAlex ids.
The pool holds 49,004 works with an id under 10^8, 28,148 of them carrying a DOI.
Measured against CrossRef, 50 drawn at random from those 28,148:
| | match | mismatch | unresolvable |
| --- | --- | --- | --- |
| id < 10^8 (n=50) | 16 | **31** | 3 |
| id ≥ 10^8, control (n=50) | 33 | 0 | 15 |
So roughly two thirds of that class is cross-wired — on the order of 18,000 pool works.
The control's 15 unresolvable are DataCite/OSF DOIs CrossRef does not hold, and its two
apparent mismatches are supplementary-file records whose CrossRef title is `Image_2.pdf`.
No genuine mismatch was found outside the low-id class.
## What reaches the export
Thinly, but it does. 6 of the 2,279 rows in the current `data/extracted.csv` have a work
id under 10^8; 4 of them carry a DOI, and all four were checked:
| `oa_work_id_r` | `doi_r` | what the DOI actually is | verdict |
| --- | --- | --- | --- |
| `W22986889` | `10.1007/s11892-012-0315-2` | "Complementary and Alternative Medicine in Diabetes Care" | **wrong** |
| `W23811235` | `10.1016/j.mce.2013.06.025` | a ponatinib inhibitor paper | **wrong** |
| `W29234848` | `10.1080/02796015.1986.12085213` | the paper itself | correct |
| `W71802120` | `10.1080/00222216.1977.11970318` | the paper itself | correct |
`pair_id` is `md5(doi_r + "|" + doi_o)`, so such a row imports under an identity built
from another paper's DOI rather than being caught at import.
## What a fix costs, which is the open question
The check itself is the one `shared/doi_verify.py` already performs for `doi_o`: fetch
what the DOI resolves to and compare title/authors. Where to put it is a cost question,
and the options differ by orders of magnitude:
1. **Every extracted row.** One CrossRef lookup per row, no OpenAlex spend (a DOI lookup
is a single-entity call and free). Simplest, and pays for ~2,300 rows to catch ~2.
2. **Only rows whose work id is under 10^8.** 6 rows of the current export. The
signal is empirical rather than structural — OpenAlex documents no such boundary —
so it should be a cheap PRE-filter for a check that still verifies, not a rule that
decides anything by itself.
3. **At Stage 2, over the admitted piles.** Catches the row before Stage 3 spends the
ladder on it, at the cost of a lookup per admitted work.
Option 2 inside Stage 3's existing verification seam looks like the right trade: the same
CrossRef call, on a population two orders of magnitude smaller, at the point where the
row is already being verified. What a mismatch should DO is the second question — the
honest answer is probably not "correct the DOI" but "flag the row and set it aside",
since a record whose metadata is cross-wired cannot be trusted to name its own paper.
## How to reproduce
```python
import csv
rows = list(csv.DictReader(open("data/extracted.csv", encoding="utf-8-sig", newline="")))
low = [r for r in rows
if (r.get("oa_work_id_r") or "").strip().lstrip("W").isdigit()
and int((r["oa_work_id_r"]).strip().lstrip("W")) < 100_000_000]
```
Then check each `doi_r` against `https://api.crossref.org/works/` and compare the
returned title with `title_r`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with shared/doi_verify.py and the Stage 3 verification seam, then reproduce the low-id cases from data/extracted.csv using CrossRef. Compare doi_r metadata with title_r and authors, and resolve how mismatches should be flagged or set aside before implementing the chosen scope. Done means the affected doi_r records are verified and mismatches cannot enter the export unchecked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100