internetarchive / internetarchive/openlibrary
Cover Service Improvements
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
## Tasks
See [2023 yearly planning notes](https://docs.google.com/document/d/1c3oxsRaepcLUxJBmgRhQl336KNSUmf5tRjUx9KSROI4/edit#):
* [x] Fix amz cover imports
* #7244
* #7316
* #7340
* [x] #7257
* [x] #8278
* Related to #6822 and #7691
* [ ] #7691 Extend the BWB monthly importer cron to [archive](https://github.com/internetarchive/olsystem/blob/master/bin/bwb_etl/process_partner_data.sh#L54-L103) & [import covers](https://github.com/internetarchive/openlibrary-bots/tree/master/BWBCoverBot): #6706
* This depends on / requires space on the host via #7257
* [ ] #7211: Task to investigate performance opportunities
**Bonus opportunities**
* [ ] Create monthly dumps which map a cover ID to isbn/lccn/olid -> access url; e.g. archive.org zip offset
* [ ] Add author cover urls to solr via #7659
## @cclauss's Notes
### Describe the problem that you'd like solved
* What documentation do we have about book cover images?
* What book cover image cron jobs do we currently run?
* Are we properly backing up the book cover images that we get from affiliates?
* Can we restore our book cover images from those backups?
* Do we provide our clients a covers API?
* If we have an API for accessing book cover images, how often is it used?
* If we have an API for accessing book cover images, how could it be improved?
[ol-covers on Sentry](https://sentry.archive.org/organizations/ia-ux/issues/?project=10)
### Proposal & Constraints
Pull request #6706 -- Add book cover images from Better World Book imports
Steps:
1. Create a dictionary of all book cover images that have been imported from Better World Books so far this year.
2. Create a dictionary of all Open Library ISBNs in the previous dictionary that do not have a book cover image.
3. Enhance the Open Library record of all ISBNs that appear in the second dictionary to add the image to the OL record.
### Additional context
### Stakeholders
---
# Initial timeline:
```
2022-08-30 Present the draft roadmap on a community call to solicit feedback.
2022-09-15 Send out a survey of needs and ideas to Librarians, Developers, Operations
2022-09-30 Release final roadmap and present on a community call
```
---
# Open Library: Roadmap for Book Cover Images
## Summary
Workstreams: Covers acquisition, storage, retrieval, and documentation
## How do we get cover images?
Open Library obtains book cover images via the covers service which is a cron job that runs in a dedicated Docker container.
## How do we store cover images?
Book cover images via the covers service are stored in `ol-home0`'s `/1/var/tmp/imports` directory in YYYY-MM.zip files. Each file is named isbn_13.jpg (e.g.: 9780425030134.jpg)
## How do we provide access to cover images?
The work on #6706 has made it clear that a low-level API that rapidly retrieves cover images based on isbn_13s would:
1. simplify the creation of advanced search on above, and
2. simplify integration into other systems.
Contributor guide
Assessment
This issue has not been assessed yet.