internetarchive / internetarchive/openlibrary-client

Flagging Edition constraint violation for automated or manual split

Open
#146 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
506
Forks
110
PR merge metrics
No merged PRs in 30d

Description

A data clean-up task which I have not yet found an open issue addressing is developing and checking constraints for unique identifiers / unique values:

This specifically targets all OL*M values that have more than one ISBN of the same length ie two ISBN13's existing on an Edition. Given the ISBN should be unique (except in the case of reprints) any Edition with two of the same length are actually conflating two editions and should be split.

Example issue: [OL23158941M](https://openlibrary.org/books/OL23158941M/Managing_water_resources_in_a_time_of_global_change?v=7) has two ISBN13 values and two Goodreads identifier from the initial import and as such has conflated two formats of a work.

Addressing this by creating a flagged task list of works that cannot be safely split automatically due to certain metadata feidls being present we can create specific work queues and flag so bots don't add metadata until the split is made to avoid further conflating the record.

ie ImportBot should check if an existing identifier exists for a work and not add a second ISBN10 if another ISBN10 exists etc.

MARC records imported with more than one ISBN of the same format should be flagged for split or not imported until a manual split/assessment is performed.

This edition needs to be split to two editions one per ISBN13 and the subsequent metadata to be split as well.

Proposed next steps:
1. Run an analysis across the editions for duplicate ISBNs of the same length to understand the scope of the issue.
2. Identify where the duplication occurred (to identify where to prevent this from occurring in the future)
3. Identify which Editions have changes made after the conflation occurred and what types of changes were made to what fields as possible down stream affect of the conflated record.
4. Identify what metadata can safely be split and what should trigger a manual review.
5. Build a list or set of lists for each split rule violated to be addressed (CSV for automated processing and public list for manual split tasks to describe what people can do to help improve the data quality in that list before it is removed. - one could argue that a specific interface should differentiate this from a reading list, but it would be easy to use the existing reading list structure to test the idea and see if people would take up the focused task for data quality.
6. Reassess if any of the previous steps still indicate areas where data quality will return and create itemized tasks and components that need to be addressed. ie block import bots from editing conflated editions, add constraints to field that should be unique, etc.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.