forrtproject / forrtproject/flora-extractor
Merge DOI-less duplicate work ids before the next validation import
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 4
Description
The shipped same-DOI alias merges (#193, filter/spec/aliases.json) cannot reach duplicate OpenAlex work records that carry NO DOI — the pool holds the same paper under several work ids distinguishable only by (normalised) title. Roughly 54 such groups sat in the 2026-08-13 export.
This got more urgent with the pair_id fix (a8771be): pair_id now falls back doi_r → OpenAlex id → title, so each DOI-less copy of the same paper gets its OWN pair_id. Before the fix the import silently kept one arbitrary copy; now every copy would import as a separate row. One same-DOI pair also remains in the export (10.17605/osf.io/fwhbu under W2949883299 + W7028064024 — sanity_check's "duplicate pair_ids: 2").
Ask: extend the alias merging to DOI-less title-duplicates (normalised-title match within the pool, HTML entities unescaped — one observed group differs only by "&", "&" and "&"), or dedupe at export by (normalised title_r, doi_o). Until it lands, the validation handoff should dedupe on import.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with filter/spec/aliases.json and the alias-merging path, then trace pair_id handling after commit a8771be through the validation import and export. Check sanity_check's duplicate pair_ids output and the cited DOI-less title variants. Done means duplicate OpenAlex records collapse reliably, including HTML-entity variants, without separate imported rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100