internetarchive / internetarchive/openlibrary
Book Availability in Solr
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
Right now, when Open Library wants to find the availability of a book, we hit Archive.org's Bulk Availability Service (which doesn't currently do much cacheing). The performance implications aren't great for either party.
## Question
Is there any internal changes API for loans which can report: "what loans have been made in the last ~1 minute"?
## Rational
Why 1 min? Back of napkin numbers; If we somehow reached 100M loans per year (way beyond what we do today) , that would be about 200 loans a minute (on avg; discounts temporal locality / popularity spikes). This number is cheap enough to calculate and process at that time increment.
## Opportunity / Benefits
With such a `loans/changes` API in place, we could:
* add accurate availability into Open Library solr and hopefully drastically lower OL dependency on IA ES @ximm
* We may go from making 10s of thousands of requests to ES --> a few requests to this API / minute.
* significantly improve performance & ux of Open Library pages which make several Bulk Availability calls to IA
* search, carousels, books, my books
## Risks & Mitigations
1. Would this change but too much stress on solr-updater? Editing 1k bulk solr records each minute or so? @cdrini
* ?
2. How do we handle cold-starts? (e.g. a solr-reindex which has no knowledge of anonymized active loans)
* On OL Books Page & borrow action, we already hit ground truth loan API. We could use this to update availability_status in solr
* We could use the Bulk Availability API to warm cache (just for records OL solr records with no availability_status)
### Stakeholders
@bfalling @ximm @mekarpeles
Contributor guide
Assessment
This issue has not been assessed yet.