internetarchive / internetarchive/openlibrary
Determine the specifics of author strong identifier matching
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Question
I have an open project about [importing books from Wikisource](https://github.com/internetarchive/openlibrary/pull/9674/). My import script uses both the Wikidata API as well as the Wikisource API to fetch as much rich information about each book as possible.
While I was developing this script, I learned about the strong identifiers Wikidata offers for authors (like VIAF id, Bookbrainz id, etc). As a proof of concept, I updated my script to include those identifiers in the import records it outputs, and then I modified the import API pipeline to match incoming books to existing authors based on those identifiers. It works, but there's just not much existing data to match to.
Before committing to this change, we should fill out those identifiers for all of OL's existing authors so that the import pipeline can actually use them for matching authors in incoming records. As Wikidata offers that information, and we already know how to get it, we should have a script that can do that backfill.
We should discuss specifics here, such as which IDs (out of [this list](https://github.com/internetarchive/openlibrary/blob/844cd36dec6dc46346c6285f098f25bd3cd2a5a5/openlibrary/plugins/openlibrary/config/author/identifiers.yml)) should be used for import matching at all (and in which priority) and how to handle conflict resolution. (Also, how do MARC records for authors factor into this?)
### Stakeholders
@RayBB @cdrini
Contributor guide
Assessment
This issue has not been assessed yet.