forrtproject / forrtproject/fred-data
FLoRA dedup key assumes every original has a DOI — DOI-less originals collide
- Dominant language
- HTML
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## The issue
The README describes the FLoRA pipeline as "deduplicated by original-replication/reproduction DOI pairs". That key is undefined when the **original** has no registered DOI — which is the normal case for books, book chapters, and pre-DOI-era papers.
If a missing `doi_o` collapses to an empty string in the key, then every DOI-less original of a given replication becomes the same pair, and a replication targeting two such originals loses one on merge. If instead the row is dropped for lacking a key, the loss is silent in a different way.
I have not verified which of those happens here — flagging the assumption rather than reporting a confirmed bug. Worth a look at the dedup step and at anything building `https://doi.org/{doi_o}` for display.
## Context from flora-extractor
The extractor is adding first-class support for DOI-less originals. Agreed representation, with no schema change (no new or renamed columns):
```
doi_o = ""
doi_o_verification = "no_doi"
oa_work_id_o = "W2003152982"
title_o = "Gender Advertisements"
```
A live example is a content-analysis replication (`doi_r = 10.25669/9kzj-tc3j`) targeting Goffman's *Gender Advertisements*. The extractor had previously linked it to a DOI that is really a jointly-catalogued record of a *review* of the book — a wrong link that passed title verification because no real DOI existed to find. These rows were blocked from export until now, so they have probably never reached this pipeline.
The extractor's own identity key, `pair_id`, falls back to `oa:` and then to a normalised title when `doi_o` is blank, and is unchanged for every row that does have a `doi_o`. If a shared fallback is wanted, `oa_work_id_o` is the natural one — OpenAlex indexes DOI-less works, which is why the extractor resolves against it.
Note the Zotero Replication Checker API does DOI-hash lookups; a DOI-less original simply cannot be looked up that way, which is fine — no user has that DOI to query with either. The concern is only dedup and merge correctness.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the FLoRA dedup step and inspect any code constructing https://doi.org/{doi_o} for display. Compare behavior for two rows sharing a replication but having different DOI-less originals, using the extractor’s pair_id fallback as context. Done means DOI-less originals remain distinct through dedup and merge, with no invalid DOI display links.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100