internetarchive / internetarchive/openlibrary-bots

Bot proposal: fill missing remote_ids.wikidata on OpenLibrary authors using Wikidata P648 reverse links + entity resolution

Open
#452 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
81
Forks
67
PR merge metrics
No merged PRs in 30d

Description

## Problem

Around 67% of OpenLibrary author records have an empty `remote_ids.wikidata`. Backfilling this identity anchor also unblocks author-record deduplication — as [@tfmorris pointed out](https://github.com/internetarchive/openlibrary/issues/12887#issuecomment-4672683851) in the AgenticCommonsBot alternate_names review ([openlibrary#12887](https://github.com/internetarchive/openlibrary/issues/12887)), merging duplicates like the multiple Su Tong records is the higher-value task, and the Q-id is the strongest cross-source join key for that decision. Filling `remote_ids.wikidata` first gives the dedup effort a reliable anchor to work from.

## Proposed bot

A small bot that, for one author at a time, adds **one** `remote_ids.wikidata` value with a short edit-comment citing the evidence.

### Two candidate sources (both must survive QA)

**Source A — Wikidata already self-declares the link (P648 reverse).**
Wikidata says `Q12345 wdt:P648 "OL9999999A"`. This is a 100% identity match because Wikidata itself has committed to the pairing. The bot's job is to reflect that claim back to OL. We only apply Source A when OL's `remote_ids.wikidata` is currently empty; if OL already has a *different* Q-id on file, the candidate goes to conflict review (see below) — never overwritten.

**Source B — Multi-field entity resolution against Wikidata Q5 (humans).**
For OL authors with no P648 reverse claim, an upstream research worker matches on structured Wikidata fields:

- **name** (label / aliases) must match
- **birth date** (P569) must match (candidates with no OL `birth_date` are dropped upstream — see Scope guardrails)
- plus at least one of: **death date** (P570), **occupation / field of work** (P106 / P101), **notable works** overlap

Evidence must cite specific Wikidata fields, not paraphrases. Bad: *"matches a historian"*. Good: *"Wikidata Q12345 P31=Q5 (human), P569=1928, P106=Q201788 (historian), notable work 'Xxx' matches OL work list"*.

If the match is ambiguous — two plausible Q-ids, or single-field-only evidence — the candidate is classified `ambiguous` and dropped. The bot does **not** write low-confidence identity links.

### Scope guardrails

- **One** `remote_ids.wikidata` value per edit
- **Author pages only** (`/authors/OL...A`). No work / edition / subject edits.
- Only sets `remote_ids.wikidata` — never overwrites (skips any author whose `remote_ids.wikidata` is already populated, even with a different value; those go to human conflict review)
- Skips authors whose `birth_date` is empty upstream (identity too weak)
- Skips authors classified `ambiguous` or `not_found` by the research worker
- Append-only on the `remote_ids` map — leaves every other identifier (VIAF, ISNI, LCNAF, ...) untouched

### Frequency

Targeting **≤ 8 edits per day** total (well below polite-bot thresholds), regardless of how large the candidate pool is. Each edit is a single PUT plus 2 GETs (OL) + 1 GET (Wikidata), paced at 1.5 s between requests. Happy to tighten further if reviewers want.

## Account

Bot username: `agenticcommonsbot` — the same account referenced in [openlibrary-bots#450 / #451](https://github.com/internetarchive/openlibrary-bots/pull/451) (the AgenticCommonsBot alternate_names PR). One account, two field-scoped bots.

The S3-key authentication path is already wired and a test login + GET against a live OL author returns 200 cleanly. The PUT is the only step pending privilege grant.

## Code

Forthcoming PR will add a `wikidata_backfill/` subdirectory under the existing `AgenticCommonsBot/` directory (keeping both field-scoped bots under one account-scoped folder) with:

- `wikidata_backfill_bot.py` — single-file Python 3, stdlib only, S3-key auth via `/account/login`
- `README.md` — usage, dry-run output, evidence checklist, conflict-review handling
- `sample_proposal.json` — a real proposal covering both a Source A (P648 reverse) and a Source B (entity resolution) case
- `sample_dry_run.txt` — captured dry-run output against a live OL author (with `remote_ids.wikidata` currently empty) so reviewers can see the exact request/response shape

## Maintainer

- GitHub: [@agentic-commons-foundation](https://github.com/agentic-commons-foundation)
- Email: wiki-bot@agentic-commons.org

## Ask

Looking for a thumbs-up on the approach before opening the implementation PR, plus pointers on anything you'd like tightened. We know `remote_ids.wikidata` is an identity-anchor field, more sensitive than `alternate_names`, so we're especially interested in review of the Source B evidence bar and the conflict-review handling.

Contributor guide

Open the contributing guide

Research direction

Start with the proposed AgenticCommonsBot/wikidata_backfill/ files: wikidata_backfill_bot.py, README.md, sample_proposal.json, and sample_dry_run.txt. Review the Source A and Source B evidence rules, conflict handling, and request pacing; done means the forthcoming implementation demonstrates safe dry runs and only applies the stated guarded updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.