internetarchive / internetarchive/openlibrary

Determine the specifics of author strong identifier matching

Open
#10,029 9 comments 0 reactions 0 assignees View on GitHub
Lead: @cdrini Needs: Community Discussion Needs: Response Priority: 3 State: Blocked Type: Question
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Question

I have an open project about [importing books from Wikisource](https://github.com/internetarchive/openlibrary/pull/9674/). My import script uses both the Wikidata API as well as the Wikisource API to fetch as much rich information about each book as possible.

While I was developing this script, I learned about the strong identifiers Wikidata offers for authors (like VIAF id, Bookbrainz id, etc). As a proof of concept, I updated my script to include those identifiers in the import records it outputs, and then I modified the import API pipeline to match incoming books to existing authors based on those identifiers. It works, but there's just not much existing data to match to.

Before committing to this change, we should fill out those identifiers for all of OL's existing authors so that the import pipeline can actually use them for matching authors in incoming records. As Wikidata offers that information, and we already know how to get it, we should have a script that can do that backfill.

We should discuss specifics here, such as which IDs (out of [this list](https://github.com/internetarchive/openlibrary/blob/844cd36dec6dc46346c6285f098f25bd3cd2a5a5/openlibrary/plugins/openlibrary/config/author/identifiers.yml)) should be used for import matching at all (and in which priority) and how to handle conflict resolution. (Also, how do MARC records for authors factor into this?)

### Stakeholders
@RayBB @cdrini

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.