forrtproject / forrtproject/flora-extractor

Merge DOI-less duplicate work ids before the next validation import

Open
#199 0 comments 0 reactions 0 assignees View on GitHub
stage-2
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

The shipped same-DOI alias merges (#193, filter/spec/aliases.json) cannot reach duplicate OpenAlex work records that carry NO DOI — the pool holds the same paper under several work ids distinguishable only by (normalised) title. Roughly 54 such groups sat in the 2026-08-13 export.

This got more urgent with the pair_id fix (a8771be): pair_id now falls back doi_r → OpenAlex id → title, so each DOI-less copy of the same paper gets its OWN pair_id. Before the fix the import silently kept one arbitrary copy; now every copy would import as a separate row. One same-DOI pair also remains in the export (10.17605/osf.io/fwhbu under W2949883299 + W7028064024 — sanity_check's "duplicate pair_ids: 2").

Ask: extend the alias merging to DOI-less title-duplicates (normalised-title match within the pool, HTML entities unescaped — one observed group differs only by "&", "&" and "&"), or dedupe at export by (normalised title_r, doi_o). Until it lands, the validation handoff should dedupe on import.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with filter/spec/aliases.json and the alias-merging path, then trace pair_id handling after commit a8771be through the validation import and export. Check sanity_check's duplicate pair_ids output and the cited DOI-less title variants. Done means duplicate OpenAlex records collapse reliably, including HTML-entity variants, without separate imported rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.