internetarchive / internetarchive/openlibrary

Import Wikisource trusted book provider data

Open
#9,671 15 comments 0 reactions 1 assignee Claimed by @pidgezero-one View on GitHub
Lead: @cdrini Needs: Response Priority: 3 Theme: Trusted Book Providers Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

Followup to https://github.com/internetarchive/openlibrary/issues/8545. Currently there are 60 books in OL that have Wikisource IDs. IDs are formatted as `langcode:title` (i.e. `en:George_Bernard_Shaw`). Import Wikisource works into Open Library.

https://github.com/internetarchive/openlibrary/wiki/Developer's-Guide-to-Data-Importing

### Breakdown
- [x] @pidgezero-one Create a script which implements proposal below to get WikiSource data and coerce the data into Open Library's import format
- [x] @cdrini Verify a ~10 sample of the resulting books
- [x] Manually verify the extracted books
- [ ] Both: Run bulk import

### Proposal & Constraints

Hit English Wikisource's API and paginate through result sets of hits that fall under the "Validated texts" category: https://en.wikisource.org/w/api.php?action=query&generator=categorymembers&gcmtitle=Category:Validated_texts&gcmlimit=500&prop=categories|info|revisions&rvprop=content&rvslots=main&format=json&cllimit=max

The response includes documents that aren't books. Books are not flagged with a distinct category. We may have to also browse Wikisource's API to manually draft a list of categories that we should ignore any member of, such as Subpages (individual chapters of books), Posters, Songs, etc.

The response includes most of the work's metadata (author, year, etc) as wiki markup of the page's infobox. Consider using a library like [wptools](https://github.com/siznax/wptools) to parse it.

In the future, we will want to expand this import to support other languages besides `en.wikisource.org`, and perhaps expand beyond the `Validated texts` category, so the solution to this should be extensible.

A potential downside to how we derive Wikisource IDs is that we use the page title, which is modifiable, instead of the page's canonical ID, and that leaves us at the mercy of Wikisource's works being moved or having their names changed. This will likely be a pretty rare (if ever) occurrence, but if we ever decide we want to use canonical IDs instead, any wikisource item can be obtained with `curid`. Example: `https://en.wikisource.org/?curid=4496925` and `https://en.wikisource.org/wiki/%22Red%22_Fed._Memoirs` are the same page. (This is also an example of a page whose title may need to be URLencoded in outbound links.)

### Leads

### Stakeholders
@cdrini @pidgezero-one


Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.