ArchiveLabs / ArchiveLabs/pyopds2_openlibrary

Use OL /search/carousels.json to reduce build_home_feed from N network calls to 1

Open
#100 0 comments 0 reactions 1 assignee Claimed by @ronibhakta1 View on GitHub
Dominant language
Python
Stars
1
Forks
2
Avg merge
4h 1m
Merged PRs (30d)
1

Description

## Summary

`build_home_feed` currently fetches each carousel group via a separate call to `OpenLibraryDataProvider.search()`, which calls `_get(OL_BASE_URL + /search.json)`. With 7 carousels, this means **7 outbound HTTP calls** on every home page cache miss — from the same server IP, in a burst, consuming OL's `api_limit` rate-limit allowance.

## Proposed Solution

Once `internetarchive/openlibrary#12986` ships a `POST /search/carousels.json` endpoint, this issue tracks updating `build_home_feed` to call it once instead of N times.

### Approach (surgical — no architectural changes)

1. Extract the query-construction logic from `search()` into a private helper `_build_search_params(query, sort, limit, language, mode, media_type, access) -> dict` so it can be called without making a network request.

2. Add `_fetch_home_groups_batch(cls, groups, ...) -> list[dict] | None` that:
- Builds one params dict per group (using the new helper)
- `POST`s them all to `{OL_BASE_URL}/search/carousels.json`
- Returns the list of raw search.json responses, or `None` on any error

3. In `build_home_feed`, **try the batch path first**, fall back to the existing `ThreadPoolExecutor` path if it fails:
```python
raw_results = cls._fetch_home_groups_batch(all_groups, ...)
if raw_results is not None:
groups = [_parse_group(title, raw) for (title, _, _), raw in zip(all_groups, raw_results)]
else:
# existing parallel-individual-fetch fallback
...
```

4. The post-processing (language alignment, acquisition-option filtering, cover filtering) stays **local** — it runs on the raw docs returned by the carousels endpoint, exactly as it does today inside `search()`.

### What does NOT change

- `search()` API and all other callers
- `_home_groups_config()` — group definitions stay here
- The `opds.openlibrary.org` caching layer (it continues to cache the combined result)
- Any other route (`/search`, `/authors`, `/books`)

## Acceptance Criteria

- [ ] `build_home_feed` makes **1** network call on a cache miss (instead of 7)
- [ ] Result is functionally identical to the existing parallel path
- [ ] Fallback to N-call path works when carousels endpoint is unavailable (404 / network error)
- [ ] Unit tests: mock carousels endpoint, verify 1 request fired; mock failure, verify fallback fires
- [ ] All existing tests pass
- [ ] `OL_BASE_URL` config respected (batch URL = `{OL_BASE_URL}/search/carousels.json`)

## Dependencies

- Blocked on `internetarchive/openlibrary#12986` (carousels endpoint must ship first)

## Related

- `ArchiveLabs/pyopds2_openlibrary#99` — httpx singleton (independent, should land first)
- `ArchiveLabs/opds.openlibrary.org#38` — 429 visibility (helps confirm impact before and after)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.