internetarchive / internetarchive/fatcat
non-lowercase DOIs
- Dominant language
- Python
- Stars
- 121
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
Fatcat has a general policy that DOIs should always be normalized and stored in lower-case. It turns out this has not actually been enforced at the API level, and the `clean_doi()` helper function in Python was not normalizing to lower-case, which has resulted in many non-lower-case release entities being created, many of which are likely duplicates.
The usual DOI importers (Crossref, Datacite) did lower-case, and the lookup API also lower-cases, which has minimized the scope of the problem, but there are still on the order of 134k duplicate records:
```
zcat release_extid.tsv.gz | cut -f3 | rg '[A-Z]' | pv -l | wc -l
139964
```
Here is an example of two release entities for the same work. The Pubmed-sourced import happened first, and resulted in a release with upper-case DOI. The Crossref import happened second (same day!) with lowercase DOI:
- https://fatcat.wiki/release/search?q=%22Prevent+SARS-CoV-2+Infection+in+Danish+Mask+Wearers%22
- https://fatcat.wiki/release/t4biwhgugzfhjpiuhoq7t6u2mu
- https://fatcat.wiki/release/x6yzj4zdl5cc3mbrpewug4lm7i
Fixing this could include multiple stages:
- [x] fix `clean_doi()` in python to lower-case DOIs
- [ ] have API creation endpoint enforce lower-casing, at least for creation (eg, don't allow creation of entities if DOI is not lower-case, but don't clobber existing records)
- [x] update and/or merge existing entities
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the Python clean_doi() helper and the API creation endpoint that accepts DOI-bearing release entities. Check how normalization is currently applied and whether creation has coverage for uppercase DOIs; done means new records cannot retain non-lowercase DOIs without changing existing records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100