internetarchive / internetarchive/openlibrary
Import endpoint should allow for any (known) work identifiers
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Problem
### TL;DR
Same as https://github.com/internetarchive/openlibrary/issues/9448 but for Works :)
### A clear and concise description of what you want to happen
Being able to include any (known to OL) identifiers for the work when importing into Open Library.
Note: This is a superset of https://github.com/internetarchive/openlibrary/issues/9471 which is only concerned with Open Library identifiers. Making a separate issue due to additional considerations (see later section).
### Expected behaviour / screenshots (ex: Figma design screenshots for UI feature)
When generating a JSON blurb for import into OL, it should be possible to provide known work identifiers in it.
### Additional Context
When importing into OL you might have a variety of identifiers available that might assist in pinpointing the correct Work (if it exists in OL). E.g., if you import from Goodreads, you will have the Goodreads work id in addition to the edition id. Right now it is not possible to provide these to the import pipeline to help with identifying authors, but it could be a great help.
### Proposal & Constraints
### What is the proposed solution / implementation?
Changing the JSON schema to allow for identifiers for authors, perhaps something like,
```diff
diff --git a/olclient/schemata/import.schema.json b/olclient/schemata/import.schema.json
index 3f00e90..319c867 100644
--- a/olclient/schemata/import.schema.json
+++ b/olclient/schemata/import.schema.json
@@ -109,19 +109,12 @@
]
},
"identifiers": {
- "type": "object",
- "patternProperties": {
- "^\\w+": { "$ref": "shared_definitions.json#/string_array" }
- },
- "description": "Unique identifiers used by external sites to identify a book. Used by Open Library to link offsite.",
- "examples": [
- {
- "standard_ebooks": ["leo-tolstoy/what-is-art/aylmer-maude"]
- },
- {
- "project_gutenberg": ["64317"]
- }
- ]
+ "type": "array",
+ "items": { "$ref": "#/definitions/import_identifier" }
+ },
+ "work_identifiers": {
+ "type": "array",
+ "items": { "$ref": "#/definitions/import_identifier" }
},
"cover": {
"type": "string",
@@ -132,6 +125,21 @@
}
},
"definitions": {
+ "import_identifier": {
+ "type": "object",
+ "patternProperties": {
+ "^\\w+": { "$ref": "shared_definitions.json#/string_array" }
+ },
+ "description": "Unique identifiers used by external sites to identify a book, author, or work. Used by Open Library to link offsite.",
+ "examples": [
+ {
+ "standard_ebooks": ["leo-tolstoy/what-is-art/aylmer-maude"]
+ },
+ {
+ "project_gutenberg": ["64317"]
+ }
+ ]
+ },
"import_author": {
"type": "object",
"additionalProperties": false,
```
and then of course have the importer pipeline actually recognise the work identifier(s) and use it for matching against existing OL works.
### Is there a precedent of this approach succeeding elsewhere?
Several MusicBrainz importer scripts use identifiers from import sources to match up identifiers in MusicBrainz. E.g., [a-tisket](https://atisket.pulsewidth.org.uk/) cross-references artist identifiers from iTunes, Deezer, and Spotify with ones known in MusicBrainz to ease the import into MusicBrainz by assigning artists to already existing ones. The [Discogs importer userscript](https://github.com/murdos/musicbrainz-userscripts/?tab=readme-ov-file#discogs_importer) does the same, but also does this for Release Groups and Labels.
Granted, the import flow for MusicBrainz is _quite_ different from Open Library, but I think it still shows how being able to look up an import source’s own identifiers can greatly help in matching against the target dataset.
### Which suggestions or requirements should be considered for how feature needs to appear or be implemented?
Some considerations:
- What happens when a work otherwise perfectly matches an existing work, but…
- the provided identifier isn’t/identifiers aren’t already known?
- does work get matched regardless?
- and the identifier is/identifiers are discarded
- and the identifier is/identifiers are added to the matched work
- a new work gets created with the identifier(s) attached?
- import fails
- provided identifier(s) not known but work already has identifier(s) of the same type(s) (e.g., Goodreads id provided, but matched work already has a different Goodreads id)
- provided identifier(s) match(es) a different work
- provided identifier(s) match(es) different work_s_(!)
- Identifier(s) match(es) existing Work but any other provided data do not
And I’m sure there are plenty of other edge cases, but this is clearly more involved than just allowing OL ids (https://github.com/internetarchive/openlibrary/issues/9471)
### Leads
### Related files
Schema: https://github.com/internetarchive/openlibrary-client/blob/master/olclient/schemata/import.schema.json (though this may be moved to this repository; see https://github.com/internetarchive/openlibrary/issues/9351 for status of this)
Import logic: https://github.com/internetarchive/openlibrary/tree/master/openlibrary/plugins/importapi
### Stakeholders
### Instructions for Contributors
- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Research direction
Start with olclient/schemata/import.schema.json and the importer logic under openlibrary/plugins/importapi; review related issues 9448, 9471, and 9351 for context. Trace how import identifiers are validated and used for matching. Done means work identifiers are accepted by the schema and consistently handled by the import pipeline, including the edge cases listed in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100