ArchiveLabs / ArchiveLabs/pyopds2_openlibrary
Use OL /search/carousels.json to reduce build_home_feed from N network calls to 1
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 4h 1m
- Merged PRs (30d)
- 1
Description
## Summary
`build_home_feed` currently fetches each carousel group via a separate call to `OpenLibraryDataProvider.search()`, which calls `_get(OL_BASE_URL + /search.json)`. With 7 carousels, this means **7 outbound HTTP calls** on every home page cache miss — from the same server IP, in a burst, consuming OL's `api_limit` rate-limit allowance.
## Proposed Solution
Once `internetarchive/openlibrary#12986` ships a `POST /search/carousels.json` endpoint, this issue tracks updating `build_home_feed` to call it once instead of N times.
### Approach (surgical — no architectural changes)
1. Extract the query-construction logic from `search()` into a private helper `_build_search_params(query, sort, limit, language, mode, media_type, access) -> dict` so it can be called without making a network request.
2. Add `_fetch_home_groups_batch(cls, groups, ...) -> list[dict] | None` that:
- Builds one params dict per group (using the new helper)
- `POST`s them all to `{OL_BASE_URL}/search/carousels.json`
- Returns the list of raw search.json responses, or `None` on any error
3. In `build_home_feed`, **try the batch path first**, fall back to the existing `ThreadPoolExecutor` path if it fails:
```python
raw_results = cls._fetch_home_groups_batch(all_groups, ...)
if raw_results is not None:
groups = [_parse_group(title, raw) for (title, _, _), raw in zip(all_groups, raw_results)]
else:
# existing parallel-individual-fetch fallback
...
```
4. The post-processing (language alignment, acquisition-option filtering, cover filtering) stays **local** — it runs on the raw docs returned by the carousels endpoint, exactly as it does today inside `search()`.
### What does NOT change
- `search()` API and all other callers
- `_home_groups_config()` — group definitions stay here
- The `opds.openlibrary.org` caching layer (it continues to cache the combined result)
- Any other route (`/search`, `/authors`, `/books`)
## Acceptance Criteria
- [ ] `build_home_feed` makes **1** network call on a cache miss (instead of 7)
- [ ] Result is functionally identical to the existing parallel path
- [ ] Fallback to N-call path works when carousels endpoint is unavailable (404 / network error)
- [ ] Unit tests: mock carousels endpoint, verify 1 request fired; mock failure, verify fallback fires
- [ ] All existing tests pass
- [ ] `OL_BASE_URL` config respected (batch URL = `{OL_BASE_URL}/search/carousels.json`)
## Dependencies
- Blocked on `internetarchive/openlibrary#12986` (carousels endpoint must ship first)
## Related
- `ArchiveLabs/pyopds2_openlibrary#99` — httpx singleton (independent, should land first)
- `ArchiveLabs/opds.openlibrary.org#38` — 429 visibility (helps confirm impact before and after)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.