forrtproject / forrtproject/flora-extractor

One OSF record ships as several rows: OpenAlex work ids are not deduplicated by OSF guid

Open
#200 1 comment 0 reactions 0 assignees View on GitHub
stage-2
Dominant language
Python
Stars
2
Forks
1
Avg merge
1d 6h
Merged PRs (30d)
4

Description

**One OSF record reaches the validation import as several rows.** OpenAlex mints more
than one work for the same OSF object, and nothing in the pipeline derives identity
from the OSF guid, so each work is screened, extracted and shipped separately.

Measured on `data/extracted.csv` (2,602 rows) and release `56076eb48fda`:

| | Count |
| --- | ---: |
| OSF records in the CSV reached by >1 OpenAlex work id | 117 |
| Surplus records | 173 |
| **Rows involved** | **296 of 2,602 (11%)** |
| OSF-owned records in the worklist with >1 work id | 460 |
| Surplus work ids there | 655 |
| Of those, already covered by an alias | **0** |

They are the same record, not near-duplicates — same title, same coded outcome:

```
qp4h8 W2776696688 'Relationship of 2D:4D Ratio to the Big Five…' cannot_be_determined
qp4h8 W7070882364 'Relationship of 2D:4D Ratio to the Big Five…' cannot_be_determined
qp4h8 W7110500188 'Relationship of 2D:4D Ratio to the Big Five…' cannot_be_determined
```

**Why the existing mechanism misses them.** `filter/spec/aliases.json` holds 3,205
entries and is the right seam — aliases resolve before any state is keyed by
`work_id`, and the file's hash feeds `alias_release`, so adding entries mints a new
release and the duplicates collapse at the next route. But it encodes OpenAlex's *own*
merge relation, and OpenAlex has never merged these: it has minted several records for
one object. The `W7…` ids are recent additions alongside older `W2…` ones.

**Scope is wider than the OSF registrant.** A published article whose OA copy lives on
OSF (`10.1037/xhp0000556` → `osf.io/ebv4q`) is the same study as its OSF record and
must not be coded twice either. So the rule is *same OSF guid ⇒ same record*, and the
canonical work is the published article where one exists, then the registrant DOI, then
the URL-only record.

Note this is deliberately WIDER than `osf_identifier()`, whose exclusion of DOI-bearing
rows is about not overwriting an article's abstract with a registration template line.
That guard must stay where it is; identity is a different question and needs its own
rule.

**Plan**
1. Derive `guid -> [work ids]` over the pool, with the canonical choice above.
2. Add the mappings to `filter/spec/aliases.json` (a reviewed git change; the file's
own convention is that provenance goes in the commit message).
3. Re-route — the new `alias_release` collapses the duplicates.
4. Re-export and confirm the row count drops by the surplus.

Found while working on #196.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with filter/spec/aliases.json and the existing osf_identifier() handling, then inspect how the work pool is routed and exported. Derive OSF guid-to-work-id mappings using the stated canonical order, add the reviewed aliases, re-route, and re-export data/extracted.csv. Done means the duplicate rows collapse and the count drops by the measured surplus.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.