internetarchive / internetarchive/openlibrary-bots
Bot proposal: fill missing remote_ids.wikidata on OpenLibrary authors using Wikidata P648 reverse links + entity resolution
- Dominant language
- Python
- Stars
- 81
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Around 67% of OpenLibrary author records have an empty `remote_ids.wikidata`. Backfilling this identity anchor also unblocks author-record deduplication — as [@tfmorris pointed out](https://github.com/internetarchive/openlibrary/issues/12887#issuecomment-4672683851) in the AgenticCommonsBot alternate_names review ([openlibrary#12887](https://github.com/internetarchive/openlibrary/issues/12887)), merging duplicates like the multiple Su Tong records is the higher-value task, and the Q-id is the strongest cross-source join key for that decision. Filling `remote_ids.wikidata` first gives the dedup effort a reliable anchor to work from.
## Proposed bot
A small bot that, for one author at a time, adds **one** `remote_ids.wikidata` value with a short edit-comment citing the evidence.
### Two candidate sources (both must survive QA)
**Source A — Wikidata already self-declares the link (P648 reverse).**
Wikidata says `Q12345 wdt:P648 "OL9999999A"`. This is a 100% identity match because Wikidata itself has committed to the pairing. The bot's job is to reflect that claim back to OL. We only apply Source A when OL's `remote_ids.wikidata` is currently empty; if OL already has a *different* Q-id on file, the candidate goes to conflict review (see below) — never overwritten.
**Source B — Multi-field entity resolution against Wikidata Q5 (humans).**
For OL authors with no P648 reverse claim, an upstream research worker matches on structured Wikidata fields:
- **name** (label / aliases) must match
- **birth date** (P569) must match (candidates with no OL `birth_date` are dropped upstream — see Scope guardrails)
- plus at least one of: **death date** (P570), **occupation / field of work** (P106 / P101), **notable works** overlap
Evidence must cite specific Wikidata fields, not paraphrases. Bad: *"matches a historian"*. Good: *"Wikidata Q12345 P31=Q5 (human), P569=1928, P106=Q201788 (historian), notable work 'Xxx' matches OL work list"*.
If the match is ambiguous — two plausible Q-ids, or single-field-only evidence — the candidate is classified `ambiguous` and dropped. The bot does **not** write low-confidence identity links.
### Scope guardrails
- **One** `remote_ids.wikidata` value per edit
- **Author pages only** (`/authors/OL...A`). No work / edition / subject edits.
- Only sets `remote_ids.wikidata` — never overwrites (skips any author whose `remote_ids.wikidata` is already populated, even with a different value; those go to human conflict review)
- Skips authors whose `birth_date` is empty upstream (identity too weak)
- Skips authors classified `ambiguous` or `not_found` by the research worker
- Append-only on the `remote_ids` map — leaves every other identifier (VIAF, ISNI, LCNAF, ...) untouched
### Frequency
Targeting **≤ 8 edits per day** total (well below polite-bot thresholds), regardless of how large the candidate pool is. Each edit is a single PUT plus 2 GETs (OL) + 1 GET (Wikidata), paced at 1.5 s between requests. Happy to tighten further if reviewers want.
## Account
Bot username: `agenticcommonsbot` — the same account referenced in [openlibrary-bots#450 / #451](https://github.com/internetarchive/openlibrary-bots/pull/451) (the AgenticCommonsBot alternate_names PR). One account, two field-scoped bots.
The S3-key authentication path is already wired and a test login + GET against a live OL author returns 200 cleanly. The PUT is the only step pending privilege grant.
## Code
Forthcoming PR will add a `wikidata_backfill/` subdirectory under the existing `AgenticCommonsBot/` directory (keeping both field-scoped bots under one account-scoped folder) with:
- `wikidata_backfill_bot.py` — single-file Python 3, stdlib only, S3-key auth via `/account/login`
- `README.md` — usage, dry-run output, evidence checklist, conflict-review handling
- `sample_proposal.json` — a real proposal covering both a Source A (P648 reverse) and a Source B (entity resolution) case
- `sample_dry_run.txt` — captured dry-run output against a live OL author (with `remote_ids.wikidata` currently empty) so reviewers can see the exact request/response shape
## Maintainer
- GitHub: [@agentic-commons-foundation](https://github.com/agentic-commons-foundation)
- Email: wiki-bot@agentic-commons.org
## Ask
Looking for a thumbs-up on the approach before opening the implementation PR, plus pointers on anything you'd like tightened. We know `remote_ids.wikidata` is an identity-anchor field, more sensitive than `alternate_names`, so we're especially interested in review of the Source B evidence bar and the conflict-review handling.
Contributor guide
Research direction
Start with the proposed AgenticCommonsBot/wikidata_backfill/ files: wikidata_backfill_bot.py, README.md, sample_proposal.json, and sample_dry_run.txt. Review the Source A and Source B evidence rules, conflict handling, and request pacing; done means the forthcoming implementation demonstrates safe dry runs and only applies the stated guarded updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100