internetarchive / internetarchive/openlibrary
Extend BWB Monthly Cron to archive + import covers
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
Related to #6822
### Describe the problem that you'd like solved
NB: Some of the details of this issue are intentionally internal as they deal with partner data sources.
Presently, every mid-month, we receive certain book data from BWB which we archive and then import. As part of this [process](https://github.com/internetarchive/olsystem/blob/master/bin/bwb_etl/process_partner_data.sh#L54-L103), we download book covers, but unlike other book data, these covers reside on disk and (a) have not yet been archived and (b) have not yet been imported.
The purpose of this issue is to **extend** the BWB mid-monthly importer cron job so that when the cron runs:
* [ ] Covers are [archived]((https://github.com/internetarchive/olsystem/blob/master/bin/bwb_etl/process_partner_data.sh#L54-L103)) in archive.org
* There's pre-existing code in the cron to archive book data, we'll need a separate block with similar logic for covers. This is likely to be the easy step.
* [ ] Covers are [imported](https://github.com/internetarchive/openlibrary-bots/tree/master/BWBCoverBot) / #6706 into Open Library.
* Importing these records depends on sufficient free space space on the host via #7257
* [ ] Provide internal `olsystem` documentation / README
## Import strategy
We've already tested that BWBCoverBot/main.py works for importing these covers. To reduce overhead, we want to **skip covers we have already imported**. Before we run the cover importer, we will first use the open library monthly editions dump to get a normalized set of isbn13 for all editions that already have covers present (see cover_id field) → add this to a sqlite table of "don't need" for running.
### Stakeholders
@bfalling @cdrini @cclauss @jimchamp
Contributor guide
Research direction
Start with the BWB cron flow in olsystem/bin/bwb_etl/process_partner_data.sh and the tested importer in BWBCoverBot/main.py. Review the monthly editions dump and the skip-record strategy, along with dependency #7257. Done means covers are archived, already-imported covers are skipped, remaining covers are imported, and internal olsystem README documentation is provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell, sqlite
- Domain
- data-engineering, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100