internetarchive / internetarchive/openlibrary
Index Lenny instance(s) holdings in Open Library
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Feature Request
In order to allow for searching of a specific lenny instance or across all lenny instances, we want to be able to store the holding in Open Library's solr.
### Proposal
```
POST /api/lenny/synchronize?check=true
> Response {
"status": "ok",
"last_synchronization": "2024-01-02T12:00:00Z"
}
POST /api/lenny/synchronize
> Do the actual synchronization
> Body {
"catalog": [
"/books/OL123M",
]
}
> Response {
"added": [ ... ],
"removed": [ ... ],
"missing": [ ... ]
}
```
1. Lenny on startup/every 30 minutes (with jitter!) does a heartbeat, and gets back the last time OL updated its catalog for this lenny instance.
- Two ways: OL can store this timestamp in infogami/db (?)
2. If the returned timestamp is less than its internal last updated timestamp, it will send its entire catalog for synchronization.
3. When OL receives a synchronize request, it will query solr to find all editions with `lenny_host:{host}`, intersect that against the provided catalog and add missing using a partial `add-distinct` update to `lenny_host`, and delete no longer supplied catalog values.
- Note: We ideally also want the ebook-access; blocked on #11264
Full reindex will delete all `lenny_host:{host}` values for editions, which will cause the next lenny heartbeat to send its catalog in entirety.
When a solr document is reindexed normally, we will copy-forward the existing `lenny_host:{host}` values from the existing solr document to the new one.
Risks:
- It's expensive to synchronize, since no partial updates are easily possible
- Race-conditions if multiple synchronize calls happen at once
### Breakdown
### Related files
Refer to [this map of common Endpoints](https://github.com/internetarchive/openlibrary/wiki/Endpoints):
*
### Requirements Checklist
Checklist of requirements that need to be satisfied in order for this issue to be closed:
* [ ]
### Stakeholders
@mekarpeles @ronibhakta1
### Instructions for Contributors
* **Before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) or pushing up changes to a PR, please first [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Assessment
This issue has not been assessed yet.