internetarchive / internetarchive/openlibrary

Video Tutorial of Implementing a Trusted Book Provider (e.g. Cita Press)

Open
#8,462 4 comments 0 reactions 2 assignees Claimed by @scottbarnes View on GitHub
Lead: @cdrini Module: Import Needs: Help Priority: 3 Theme: Trusted Book Providers Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Background

OpenLibrary.org is a catalog of every book published and where possible, it links to sources where you can access books to read or borrow.

Many of these books are fulfilled by the Internet Archive's book lending library program. In addition, Open Library links to many vetted partner book sources like Project Gutenberg, Librivox, Standard Ebooks, OpenStax, and others through what it calls its Trusted Book Providers (TBP) program.

We have a [form](https://docs.google.com/forms/d/e/1FAIpQLSdufB9B1aX9RceKLycbxJBW9bv9EJv_PEiVNu8cXycXdD134A/viewform) where organizations can apply to be considered for the TBP program.

Currently, each TBP is added to the website manually and this involves a few steps.
1. Writing a python script to load the metadata from the partner into a format that can be imported into Open Library
2. Extending the Open Library TBP code to register a new source
3. Using the openlibrary-client with step [1] to submit these records for import

The program is detailed here: https://openlibrary.org/trusted-book-providers. There's also a blog post [here](https://blog.openlibrary.org/2021/12/20/introducing-trusted-book-providers/) with more details.

### Describe the problem that you'd like solved

Something that would be really helpful is a video recording of implementing a Trusted Book Provider. It should only take 1h (assuming the data is all in one place). Citapress may be a good place to start. Once we have a single example, others should be able to relatively easily add new sources which could make a big impact.

We have permission from Cita to pursue and integration and all their data and books are available from citapress.org and http://citapress.org/page-data/index/page-data.json

### Additional context

Here's a script which fetches the publisher's catalog and begins to map Cita press's books (http://citapress.org/page-data/index/page-data.json) to Open Library's import schema (https://github.com/internetarchive/openlibrary-client/blob/master/olclient/schemata/import.schema.json):

```python
import requests

r = requests.get('https://citapress.org/page-data/index/page-data.json')
data = r.json() # we load the fetched website data into python as json which we can manipulate like a dictionary
books = data['result']['data']['allMarkdownRemark']['nodes']

# We need these books to be converted into this format: https://github.com/internetarchive/openlibrary-client/blob/master/olclient/schemata/import.schema.json
for book in books:
openlibrary_book = {
'title': book['frontmatter']['title'],
'description': book['frontmatter']['description'],
'cover': '???',
"source_records": 'citapress', # ignore, leave as is :)
"authors": [], # might be missing from the data!
"publishers": [],
"publish_date": ''
}
print(openlibrary_book)
```

### Examples & References
Here's an example of importing...

* Pressbooks: https://github.com/internetarchive/openlibrary/pull/6582/files
* Project Gutenberg, Librivox, & Standard Ebooks: https://github.com/internetarchive/openlibrary/pull/5271/files
* OpenStax: https://github.com/internetarchive/openlibrary/pull/6138/files

### Criteria

This issue may be closed when we have:
1. a video recording end-to-end (hackathon style) of adding a new Trusted Book Provider
2. when we've added a provider PR for Cita
3. when there's additionally some wiki document/tutorial which explains the process, references PR examples, etc (see: the examples listed in https://openlibrary.org/trusted-book-providers)

### Stakeholders

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.