internetarchive / internetarchive/openlibrary

Bot Account Application: extend AgenticCommonsBot scope to remote_ids.wikidata backfill

Open
#13,136 4 comments 0 reactions 0 assignees View on GitHub
Lead: @mekarpeles Needs: Response Needs: Staff Decision Priority: 3 Theme: Bots
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

## Bot Account Application Request (Scope Extension)

**Account Name**: AgenticCommonsBot (existing account, no new account requested)
**GitHub Username**: WendyLee07 (working as part of the [Agentic Commons](https://agentic-commons.org) project)
**Purpose**: Extend the existing AgenticCommonsBot scope to also backfill missing `remote_ids.wikidata` on OpenLibrary author records, using Wikidata P648 reverse claims and multi-field entity resolution as evidence.

### Request

This is a **scope extension** for the same `agenticcommonsbot` account whose original application is [#12887](https://github.com/internetarchive/openlibrary/issues/12887) (alternate_names). We're keeping one bot account and adding a second field-scoped write path under it.

- Please grant / retain bot privileges on the `AgenticCommonsBot` account
- Please confirm the "API" usergroup grant covers `remote_ids.wikidata` PUTs (same endpoint, different field)

### Why this matters

Around 67% of OL author records have an empty `remote_ids.wikidata`. Backfilling this identity anchor also unblocks author-record deduplication — as [@tfmorris pointed out](https://github.com/internetarchive/openlibrary/issues/12887#issuecomment-4672683851) during the alternate_names review, merging duplicates like the multiple Su Tong records is the higher-value task, and the Q-id is the strongest cross-source join key for that decision. Filling `remote_ids.wikidata` first gives the dedup effort a reliable anchor to work from.

### Bot behavior (per edit)

For one author at a time:
1. `GET /authors/OL...A.json`
2. Verify `remote_ids.wikidata` is still empty (skip if already populated — never overwrite)
3. `GET Wikidata Special:EntityData/.json` — re-validate the entity exists, is a Q5 human, and the evidence claims (P648 for Source A; P569/P570/P106 for Source B) are still present
4. Append `remote_ids.wikidata = "Q..."` (all other identifiers untouched)
5. `PUT` the updated record with a factual-citation edit comment
6. Verify via re-fetch GET

### Scope guardrails

- **Author pages only** (`/authors/OL...A`). No work / edition / subject edits.
- **Append-only** — never overwrites the `remote_ids.wikidata` value or touches other `remote_ids` entries (VIAF, ISNI, LCNAF, ...).
- Skips authors with `remote_ids.wikidata` already populated (goes to human conflict review, never to the bot).
- Skips authors with empty `birth_date` on OL side (identity too weak for name-only matching).
- **≤ 8 edits/day total**, well below polite-bot thresholds.

### Evidence requirement

Two candidate paths, both audited upstream before the bot sees the proposal:

- **Source A (P648 reverse):** Wikidata already declares `wdt:P648 = OLID`. The bot re-fetches the Wikidata entity and confirms the P648 claim is still present pointing at the same OLID. Reflects a link Wikidata has already committed to.
- **Source B (entity resolution):** name + birth date (required) + at least one of {death date, occupation P106/P101, notable works overlap} matched against Wikidata Q5 (humans). Evidence must cite specific Wikidata field IDs (e.g. `"Wikidata Q12345 P569=1928"`), not paraphrases.

Ambiguous or single-field-only candidates are dropped upstream — no speculative or low-confidence writes.

### Edit comment format

Pure factual citation, no slogans or project attribution.

Source A example:
> Adding remote_ids.wikidata=Q778276 per Wikidata P648 cross-reference (https://www.wikidata.org/wiki/Q778276).

Source B example:
> Adding remote_ids.wikidata=Q778276. Wikidata Q778276 P31=Q5 (human), P569=1963-01-23 matches OL birth_date=1963-01-23, P106=Q49757 (novelist) matches OL author bio.

### Pipeline architecture

Same shape as the alternate_names pipeline in #12887:
1. Streaming filter over the monthly OL author dump (`ol_dump_authors_latest.txt.gz`) — no OL crawling.
2. Research worker gathers structured evidence and produces a proposal (`found` / `ambiguous` / `not_found`).
3. **Independent QA gate** re-fetches Wikidata and verifies the QID exists, is Q5, and the evidence claims are still present before the bot sees the proposal.
4. Only `found` proposals with high-confidence evidence reach the bot, which performs the single PUT and post-verify GET.

The bot's input is a structured proposal whose QID it re-fetches from Wikidata at PUT time. `remote_ids.wikidata` is an identity anchor — writing the wrong Q-id fuses two authors together — so the review bar is deliberately higher than for `alternate_names`.

### Code

Filed at [openlibrary-bots#453](https://github.com/internetarchive/openlibrary-bots/pull/453) — single-file Python 3 stdlib script in `AgenticCommonsBot/wikidata_backfill/`, with a real proposal ([Liu Xiaobo → Q41617 for OL4155123A](https://openlibrary.org/authors/OL4155123A)) and captured dry-run output. Login HTTP 200 and GET 200 already verified against the live bot account; PUT pending this grant.

Related: [openlibrary-bots#452](https://github.com/internetarchive/openlibrary-bots/issues/452) (design discussion).

### Operator contact

- GitHub: [@WendyLee07](https://github.com/WendyLee07)
- Email: wiki-bot@agentic-commons.org

Thank you for reviewing! 🙏

@mekarpeles @hornc

Contributor guide

Open the contributing guide

Research direction

Review the existing scope request in #12887, the single-file Python bot under AgenticCommonsBot/wikidata_backfill/ in openlibrary-bots#453, and the design discussion in openlibrary-bots#452. Confirm that the proposed author-only, append-only Wikidata PUT flow and evidence checks fit the requested API usergroup grant; done means the scope decision and permission coverage are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.