internetarchive / internetarchive/openlibrary

Create Project Gutenberg importer

Open
#11,468 11 comments 0 reactions 0 assignees View on GitHub
Lead: @cdrini Module: Import Needs: Response Priority: 3 Theme: Trusted Book Providers Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Feature Request

Classics are some of the most used books on Open Library, as exemplified by the frequent occurrence of these books in the trending carousel. Many patrons are looking for easy-to-read formats for e-readers and phones, and Project Gutenberg is a fantastic open source and open collaboration initiative that provides these books.

We currently have [~1.4k project gutenberg editions in Open Library](https://openlibrary.org/search?q=id_project_gutenberg%3A*+ia%3A*&mode=everything), that have been manually added by librarians/patrons. Including records and read links for the over 75k editions available in Project Gutenberg would help make this resource more discoverable as well help patrons find accessible copies of books they are looking for.

Creating an importer that we can hook up to https://openlibrary.org/import/preview would be a good first step, that let's us work through the logic of how to fetch/marshal the data in the correct import format.

### Data sources

- We do have a backup of Project Gutenberg books in the Archive. Here's a sample record: https://archive.org/details/tomsawyerdetecti00093gut
- This collection also contains a big listing of Gutenberg books, but not sure if it's complete or up-to-date or has/does not have duplicates, etc: https://archive.org/details/gutenberg
- Maybe their official website has a data dump? https://www.gutenberg.org/
- Each gutenberg ebook has an ID number, eg `93`
- html page url: https://www.gutenberg.org/ebooks/93
- Archive page: can maybe be found by this query: https://archive.org/search?query=call_number%3A%22gutenberg+etext%23+93%22
- Each ebook has a "zip" folder of files, with I think no consistent structure/naming: https://www.gutenberg.org/files/93/

* Phase 1: Given an gutenberg ebook ID, fetch metadata (from somewhere) and create the Open Library ImportRecord format, that can be imported via `/import/preview`.
* Phase 2: Look around for a bulk metadata dump? That we can use to potentially either just get a large number of ebook IDs and then use the process from phase 1, or ideally, have a bulk metadata dump that we can import directly.

### Breakdown

### Related files

Refer to [this map of common Endpoints](https://github.com/internetarchive/openlibrary/wiki/Endpoints):
* The `/import/preview` endpoint: https://github.com/internetarchive/openlibrary/blob/ffddccf5c88b3f3549f8870f512b7f3d0acaf7ed/openlibrary/plugins/importapi/import_ui.py

### Requirements Checklist
Checklist of requirements that need to be satisfied in order for this issue to be closed:
* [ ]

### Stakeholders

* @pidgezero-one


### Instructions for Contributors

* **Before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) or pushing up changes to a PR, please first [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.