internetarchive / internetarchive/openlibrary

Extend BWB Monthly Cron to archive + import covers

Open
#7,691 1 comment 0 reactions 0 assignees View on GitHub
Affects: Operations Lead: @mekarpeles Module: Cover Service Module: Import Needs: Breakdown Needs: Detail Priority: 2 Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

Related to #6822

### Describe the problem that you'd like solved

NB: Some of the details of this issue are intentionally internal as they deal with partner data sources.

Presently, every mid-month, we receive certain book data from BWB which we archive and then import. As part of this [process](https://github.com/internetarchive/olsystem/blob/master/bin/bwb_etl/process_partner_data.sh#L54-L103), we download book covers, but unlike other book data, these covers reside on disk and (a) have not yet been archived and (b) have not yet been imported.

The purpose of this issue is to **extend** the BWB mid-monthly importer cron job so that when the cron runs:
* [ ] Covers are [archived]((https://github.com/internetarchive/olsystem/blob/master/bin/bwb_etl/process_partner_data.sh#L54-L103)) in archive.org
* There's pre-existing code in the cron to archive book data, we'll need a separate block with similar logic for covers. This is likely to be the easy step.
* [ ] Covers are [imported](https://github.com/internetarchive/openlibrary-bots/tree/master/BWBCoverBot) / #6706 into Open Library.
* Importing these records depends on sufficient free space space on the host via #7257
* [ ] Provide internal `olsystem` documentation / README

## Import strategy

We've already tested that BWBCoverBot/main.py works for importing these covers. To reduce overhead, we want to **skip covers we have already imported**. Before we run the cover importer, we will first use the open library monthly editions dump to get a normalized set of isbn13 for all editions that already have covers present (see cover_id field) → add this to a sqlite table of "don't need" for running.

### Stakeholders

@bfalling @cdrini @cclauss @jimchamp

Contributor guide

Open the contributing guide

Research direction

Start with the BWB cron flow in olsystem/bin/bwb_etl/process_partner_data.sh and the tested importer in BWBCoverBot/main.py. Review the monthly editions dump and the skip-record strategy, along with dependency #7257. Done means covers are archived, already-imported covers are skipped, remaining covers are imported, and internal olsystem README documentation is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell, sqlite
Domain
data-engineering, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.