internetarchive / internetarchive/openlibrary
Create Project Gutenberg importer
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Feature Request
Classics are some of the most used books on Open Library, as exemplified by the frequent occurrence of these books in the trending carousel. Many patrons are looking for easy-to-read formats for e-readers and phones, and Project Gutenberg is a fantastic open source and open collaboration initiative that provides these books.
We currently have [~1.4k project gutenberg editions in Open Library](https://openlibrary.org/search?q=id_project_gutenberg%3A*+ia%3A*&mode=everything), that have been manually added by librarians/patrons. Including records and read links for the over 75k editions available in Project Gutenberg would help make this resource more discoverable as well help patrons find accessible copies of books they are looking for.
Creating an importer that we can hook up to https://openlibrary.org/import/preview would be a good first step, that let's us work through the logic of how to fetch/marshal the data in the correct import format.
### Data sources
- We do have a backup of Project Gutenberg books in the Archive. Here's a sample record: https://archive.org/details/tomsawyerdetecti00093gut
- This collection also contains a big listing of Gutenberg books, but not sure if it's complete or up-to-date or has/does not have duplicates, etc: https://archive.org/details/gutenberg
- Maybe their official website has a data dump? https://www.gutenberg.org/
- Each gutenberg ebook has an ID number, eg `93`
- html page url: https://www.gutenberg.org/ebooks/93
- Archive page: can maybe be found by this query: https://archive.org/search?query=call_number%3A%22gutenberg+etext%23+93%22
- Each ebook has a "zip" folder of files, with I think no consistent structure/naming: https://www.gutenberg.org/files/93/
* Phase 1: Given an gutenberg ebook ID, fetch metadata (from somewhere) and create the Open Library ImportRecord format, that can be imported via `/import/preview`.
* Phase 2: Look around for a bulk metadata dump? That we can use to potentially either just get a large number of ebook IDs and then use the process from phase 1, or ideally, have a bulk metadata dump that we can import directly.
### Breakdown
### Related files
Refer to [this map of common Endpoints](https://github.com/internetarchive/openlibrary/wiki/Endpoints):
* The `/import/preview` endpoint: https://github.com/internetarchive/openlibrary/blob/ffddccf5c88b3f3549f8870f512b7f3d0acaf7ed/openlibrary/plugins/importapi/import_ui.py
### Requirements Checklist
Checklist of requirements that need to be satisfied in order for this issue to be closed:
* [ ]
### Stakeholders
* @pidgezero-one
### Instructions for Contributors
* **Before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) or pushing up changes to a PR, please first [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Assessment
This issue has not been assessed yet.