internetarchive / internetarchive/openlibrary

Cover Service Improvements

Open
#6,822 1 comment 0 reactions 0 assignees View on GitHub
Lead: @mekarpeles Module: Cover Service Needs: Detail Priority: 2 Type: Epic Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

## Tasks

See [2023 yearly planning notes](https://docs.google.com/document/d/1c3oxsRaepcLUxJBmgRhQl336KNSUmf5tRjUx9KSROI4/edit#):

* [x] Fix amz cover imports
* #7244
* #7316
* #7340
* [x] #7257
* [x] #8278
* Related to #6822 and #7691
* [ ] #7691 Extend the BWB monthly importer cron to [archive](https://github.com/internetarchive/olsystem/blob/master/bin/bwb_etl/process_partner_data.sh#L54-L103) & [import covers](https://github.com/internetarchive/openlibrary-bots/tree/master/BWBCoverBot): #6706
* This depends on / requires space on the host via #7257
* [ ] #7211: Task to investigate performance opportunities

**Bonus opportunities**
* [ ] Create monthly dumps which map a cover ID to isbn/lccn/olid -> access url; e.g. archive.org zip offset
* [ ] Add author cover urls to solr via #7659

## @cclauss's Notes

### Describe the problem that you'd like solved

* What documentation do we have about book cover images?
* What book cover image cron jobs do we currently run?
* Are we properly backing up the book cover images that we get from affiliates?
* Can we restore our book cover images from those backups?
* Do we provide our clients a covers API?
* If we have an API for accessing book cover images, how often is it used?
* If we have an API for accessing book cover images, how could it be improved?

[ol-covers on Sentry](https://sentry.archive.org/organizations/ia-ux/issues/?project=10)

### Proposal & Constraints

Pull request #6706 -- Add book cover images from Better World Book imports
Steps:
1. Create a dictionary of all book cover images that have been imported from Better World Books so far this year.
2. Create a dictionary of all Open Library ISBNs in the previous dictionary that do not have a book cover image.
3. Enhance the Open Library record of all ISBNs that appear in the second dictionary to add the image to the OL record.

### Additional context

### Stakeholders

---
# Initial timeline:
```
2022-08-30 Present the draft roadmap on a community call to solicit feedback.
2022-09-15 Send out a survey of needs and ideas to Librarians, Developers, Operations
2022-09-30 Release final roadmap and present on a community call
```
---
# Open Library: Roadmap for Book Cover Images
## Summary
Workstreams: Covers acquisition, storage, retrieval, and documentation

## How do we get cover images?
Open Library obtains book cover images via the covers service which is a cron job that runs in a dedicated Docker container.

## How do we store cover images?
Book cover images via the covers service are stored in `ol-home0`'s `/1/var/tmp/imports` directory in YYYY-MM.zip files. Each file is named isbn_13.jpg (e.g.: 9780425030134.jpg)

## How do we provide access to cover images?
The work on #6706 has made it clear that a low-level API that rapidly retrieves cover images based on isbn_13s would:
1. simplify the creation of advanced search on above, and
2. simplify integration into other systems.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.