internetarchive / internetarchive/openlibrary

Open Library Solr Bulk Indexing BWB & Lenny OPDS (price)

Open
#12,138 2 comments 0 reactions 1 assignee Claimed by @cdrini View on GitHub
Lead: @mekarpeles Module: Solr Needs: Staff / Internal Priority: 2 Solr Next
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

[P2/P1] Feature: Index and harvest a curated list of OPDS feeds on a schedule, update Open Library and Solr

### Problem
Open Library needs a mechanism to index/harvest a manually curated list of OPDS URL feeds. Currently, there is no system to repeatedly check and update these feeds within Open Library and Solr.

### Solution Proposal
- Allow Open Library administrators to maintain a curated list of OPDS feed URLs
- Implement a daily cron job that:
- Checks each OPDS feed for updates (define: hash, modified date, or other)
- Detects changes since last check
- Updates Open Library and pushes changes to Solr

### Requirements
- Method to determine when an OPDS feed has changed (e.g., ETag, Last-Modified, checksum, etc.)
- Efficient strategy for update detection (should not re-harvest if unchanged)
- Update or insert new entries into Open Library and Solr index

### Open Questions
- What criteria should be used to determine if an OPDS feed has changed?
- Should this only support Atom OPDS feeds, or also RSS?
- Where to store the status/history of the last harvested state per feed?

---
**Priority:** P2/P1 (High)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.