internetarchive / internetarchive/openlibrary
Bot Account Application: extend AgenticCommonsBot scope to remote_ids.wikidata backfill
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
## Bot Account Application Request (Scope Extension)
**Account Name**: AgenticCommonsBot (existing account, no new account requested)
**GitHub Username**: WendyLee07 (working as part of the [Agentic Commons](https://agentic-commons.org) project)
**Purpose**: Extend the existing AgenticCommonsBot scope to also backfill missing `remote_ids.wikidata` on OpenLibrary author records, using Wikidata P648 reverse claims and multi-field entity resolution as evidence.
### Request
This is a **scope extension** for the same `agenticcommonsbot` account whose original application is [#12887](https://github.com/internetarchive/openlibrary/issues/12887) (alternate_names). We're keeping one bot account and adding a second field-scoped write path under it.
- Please grant / retain bot privileges on the `AgenticCommonsBot` account
- Please confirm the "API" usergroup grant covers `remote_ids.wikidata` PUTs (same endpoint, different field)
### Why this matters
Around 67% of OL author records have an empty `remote_ids.wikidata`. Backfilling this identity anchor also unblocks author-record deduplication — as [@tfmorris pointed out](https://github.com/internetarchive/openlibrary/issues/12887#issuecomment-4672683851) during the alternate_names review, merging duplicates like the multiple Su Tong records is the higher-value task, and the Q-id is the strongest cross-source join key for that decision. Filling `remote_ids.wikidata` first gives the dedup effort a reliable anchor to work from.
### Bot behavior (per edit)
For one author at a time:
1. `GET /authors/OL...A.json`
2. Verify `remote_ids.wikidata` is still empty (skip if already populated — never overwrite)
3. `GET Wikidata Special:EntityData/.json` — re-validate the entity exists, is a Q5 human, and the evidence claims (P648 for Source A; P569/P570/P106 for Source B) are still present
4. Append `remote_ids.wikidata = "Q..."` (all other identifiers untouched)
5. `PUT` the updated record with a factual-citation edit comment
6. Verify via re-fetch GET
### Scope guardrails
- **Author pages only** (`/authors/OL...A`). No work / edition / subject edits.
- **Append-only** — never overwrites the `remote_ids.wikidata` value or touches other `remote_ids` entries (VIAF, ISNI, LCNAF, ...).
- Skips authors with `remote_ids.wikidata` already populated (goes to human conflict review, never to the bot).
- Skips authors with empty `birth_date` on OL side (identity too weak for name-only matching).
- **≤ 8 edits/day total**, well below polite-bot thresholds.
### Evidence requirement
Two candidate paths, both audited upstream before the bot sees the proposal:
- **Source A (P648 reverse):** Wikidata already declares `wdt:P648 = OLID`. The bot re-fetches the Wikidata entity and confirms the P648 claim is still present pointing at the same OLID. Reflects a link Wikidata has already committed to.
- **Source B (entity resolution):** name + birth date (required) + at least one of {death date, occupation P106/P101, notable works overlap} matched against Wikidata Q5 (humans). Evidence must cite specific Wikidata field IDs (e.g. `"Wikidata Q12345 P569=1928"`), not paraphrases.
Ambiguous or single-field-only candidates are dropped upstream — no speculative or low-confidence writes.
### Edit comment format
Pure factual citation, no slogans or project attribution.
Source A example:
> Adding remote_ids.wikidata=Q778276 per Wikidata P648 cross-reference (https://www.wikidata.org/wiki/Q778276).
Source B example:
> Adding remote_ids.wikidata=Q778276. Wikidata Q778276 P31=Q5 (human), P569=1963-01-23 matches OL birth_date=1963-01-23, P106=Q49757 (novelist) matches OL author bio.
### Pipeline architecture
Same shape as the alternate_names pipeline in #12887:
1. Streaming filter over the monthly OL author dump (`ol_dump_authors_latest.txt.gz`) — no OL crawling.
2. Research worker gathers structured evidence and produces a proposal (`found` / `ambiguous` / `not_found`).
3. **Independent QA gate** re-fetches Wikidata and verifies the QID exists, is Q5, and the evidence claims are still present before the bot sees the proposal.
4. Only `found` proposals with high-confidence evidence reach the bot, which performs the single PUT and post-verify GET.
The bot's input is a structured proposal whose QID it re-fetches from Wikidata at PUT time. `remote_ids.wikidata` is an identity anchor — writing the wrong Q-id fuses two authors together — so the review bar is deliberately higher than for `alternate_names`.
### Code
Filed at [openlibrary-bots#453](https://github.com/internetarchive/openlibrary-bots/pull/453) — single-file Python 3 stdlib script in `AgenticCommonsBot/wikidata_backfill/`, with a real proposal ([Liu Xiaobo → Q41617 for OL4155123A](https://openlibrary.org/authors/OL4155123A)) and captured dry-run output. Login HTTP 200 and GET 200 already verified against the live bot account; PUT pending this grant.
Related: [openlibrary-bots#452](https://github.com/internetarchive/openlibrary-bots/issues/452) (design discussion).
### Operator contact
- GitHub: [@WendyLee07](https://github.com/WendyLee07)
- Email: wiki-bot@agentic-commons.org
Thank you for reviewing! 🙏
@mekarpeles @hornc
Contributor guide
Research direction
Review the existing scope request in #12887, the single-file Python bot under AgenticCommonsBot/wikidata_backfill/ in openlibrary-bots#453, and the design discussion in openlibrary-bots#452. Confirm that the proposed author-only, append-only Wikidata PUT flow and evidence checks fit the requested API usergroup grant; done means the scope decision and permission coverage are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100