internetarchive / internetarchive/fatcat

non-lowercase DOIs

Open
#83 1 comment 0 reactions 0 assignees View on GitHub
bug content
Dominant language
Python
Stars
121
Forks
18
PR merge metrics
No merged PRs in 30d

Description

Fatcat has a general policy that DOIs should always be normalized and stored in lower-case. It turns out this has not actually been enforced at the API level, and the `clean_doi()` helper function in Python was not normalizing to lower-case, which has resulted in many non-lower-case release entities being created, many of which are likely duplicates.

The usual DOI importers (Crossref, Datacite) did lower-case, and the lookup API also lower-cases, which has minimized the scope of the problem, but there are still on the order of 134k duplicate records:

```
zcat release_extid.tsv.gz | cut -f3 | rg '[A-Z]' | pv -l | wc -l
139964
```

Here is an example of two release entities for the same work. The Pubmed-sourced import happened first, and resulted in a release with upper-case DOI. The Crossref import happened second (same day!) with lowercase DOI:

- https://fatcat.wiki/release/search?q=%22Prevent+SARS-CoV-2+Infection+in+Danish+Mask+Wearers%22
- https://fatcat.wiki/release/t4biwhgugzfhjpiuhoq7t6u2mu
- https://fatcat.wiki/release/x6yzj4zdl5cc3mbrpewug4lm7i

Fixing this could include multiple stages:

- [x] fix `clean_doi()` in python to lower-case DOIs
- [ ] have API creation endpoint enforce lower-casing, at least for creation (eg, don't allow creation of entities if DOI is not lower-case, but don't clobber existing records)
- [x] update and/or merge existing entities

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the Python clean_doi() helper and the API creation endpoint that accepts DOI-bearing release entities. Check how normalization is currently applied and whether creation has coverage for uppercase DOIs; done means new records cannot retain non-lowercase DOIs without changing existing records.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.