internetarchive / internetarchive/openlibrary

Index Lenny instance(s) holdings in Open Library

Open
#11,384 3 comments 1 reaction 1 assignee Claimed by @mekarpeles View on GitHub
Lead: @cdrini Module: Solr Needs: Breakdown Priority: 3 Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Feature Request

In order to allow for searching of a specific lenny instance or across all lenny instances, we want to be able to store the holding in Open Library's solr.

### Proposal
```
POST /api/lenny/synchronize?check=true
> Response {
"status": "ok",
"last_synchronization": "2024-01-02T12:00:00Z"
}

POST /api/lenny/synchronize
> Do the actual synchronization
> Body {
"catalog": [
"/books/OL123M",
]
}
> Response {
"added": [ ... ],
"removed": [ ... ],
"missing": [ ... ]
}
```

1. Lenny on startup/every 30 minutes (with jitter!) does a heartbeat, and gets back the last time OL updated its catalog for this lenny instance.
- Two ways: OL can store this timestamp in infogami/db (?)
2. If the returned timestamp is less than its internal last updated timestamp, it will send its entire catalog for synchronization.
3. When OL receives a synchronize request, it will query solr to find all editions with `lenny_host:{host}`, intersect that against the provided catalog and add missing using a partial `add-distinct` update to `lenny_host`, and delete no longer supplied catalog values.
- Note: We ideally also want the ebook-access; blocked on #11264

Full reindex will delete all `lenny_host:{host}` values for editions, which will cause the next lenny heartbeat to send its catalog in entirety.

When a solr document is reindexed normally, we will copy-forward the existing `lenny_host:{host}` values from the existing solr document to the new one.

Risks:
- It's expensive to synchronize, since no partial updates are easily possible
- Race-conditions if multiple synchronize calls happen at once

### Breakdown

### Related files

Refer to [this map of common Endpoints](https://github.com/internetarchive/openlibrary/wiki/Endpoints):
*

### Requirements Checklist
Checklist of requirements that need to be satisfied in order for this issue to be closed:
* [ ]

### Stakeholders

@mekarpeles @ronibhakta1


### Instructions for Contributors

* **Before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) or pushing up changes to a PR, please first [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.