internetarchive / internetarchive/openlibrary

Data Analysis of Potential Duplicate Authors

Open
#10,438 6 comments 0 reactions 1 assignee Claimed by @maifeeulasad View on GitHub
Lead: @RayBB Needs: Help Needs: Response Needs: Review Assignee Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Proposal

[Here](https://internetarchive.slack.com/archives/C0ETZV72L/p1738991328546269?thread_ts=1738954311.514009&cid=C0ETZV72L) we have a list of 100k+ authors that have the exact same name and IDs that are only one off from each other. They are likely to be duplicate authors because of a race condition. As of 2024 there are few instances of this problem. But we still need to fix up the old instances.

Here's what someone should do:
1. For every single author get the number of works and if they only have 1 work (it seems most do) get the title of that work. Upload that here as a CSV like the one already attached but with these two additional fields.
2. From that, we should produce a list of all pairs of authors that have identical names, works with identical (case in-sensitive) titles, and IDs off by one.

From there staff can decide if we want to do an automated merge.

To work on this please use the data dumps, do not call the API.

CSV of authors in case you can't see the file on slack:

[ids_next_to_each_other.csv](https://github.com/user-attachments/files/18756768/ids_next_to_each_other.csv)

### Justification

_No response_

### Breakdown

#### Requirements Checklist

* [ ]

#### Related files

*

#### Stakeholders

*


#### Instructions for Contributors

Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.