internetarchive / internetarchive/openlibrary

Book Availability in Solr

Open
#7,450 7 comments 0 reactions 1 assignee Claimed by @benbdeitch View on GitHub
Lead: @cdrini Module: Solr Module: Solr Updater Needs: Breakdown Needs: Community Discussion Priority: 2 Theme: Book Page Theme: Public APIs Theme: Search Type: Epic Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

Right now, when Open Library wants to find the availability of a book, we hit Archive.org's Bulk Availability Service (which doesn't currently do much cacheing). The performance implications aren't great for either party.

## Question

Is there any internal changes API for loans which can report: "what loans have been made in the last ~1 minute"?

## Rational

Why 1 min? Back of napkin numbers; If we somehow reached 100M loans per year (way beyond what we do today) , that would be about 200 loans a minute (on avg; discounts temporal locality / popularity spikes). This number is cheap enough to calculate and process at that time increment.

## Opportunity / Benefits

With such a `loans/changes` API in place, we could:
* add accurate availability into Open Library solr and hopefully drastically lower OL dependency on IA ES @ximm
* We may go from making 10s of thousands of requests to ES --> a few requests to this API / minute.
* significantly improve performance & ux of Open Library pages which make several Bulk Availability calls to IA
* search, carousels, books, my books

## Risks & Mitigations

1. Would this change but too much stress on solr-updater? Editing 1k bulk solr records each minute or so? @cdrini
* ?
2. How do we handle cold-starts? (e.g. a solr-reindex which has no knowledge of anonymized active loans)
* On OL Books Page & borrow action, we already hit ground truth loan API. We could use this to update availability_status in solr
* We could use the Bulk Availability API to warm cache (just for records OL solr records with no availability_status)

### Stakeholders

@bfalling @ximm @mekarpeles

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.