internetarchive / internetarchive/openlibrary
[Performance] Audit and reduce direct calls to archive.org/metadata/{ocaid}
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
## Problem / Opportunity
Open Library calls `https://archive.org/metadata/{ocaid}` (via `ia.get_metadata()` or direct HTTP) in several places — some of which are hot paths and fetch the full item metadata payload when only a single field (`collection`) is actually used. These are expensive round-trips to archive.org, and in several cases the needed data is already available in OL's own Solr index (e.g. `ebook_access`, derived during indexing from the same IA metadata).
The goal is to identify which calls are necessary vs. redundant, and eliminate or replace those that can use local Solr data instead.
**Success criteria:** Reduce the number of cold-path `archive.org/metadata/{ocaid}` calls on page renders and the Books Read API without regression to availability display or borrowing behavior.
## Proposal
For call sites that only use `collection` to check `"inlibrary"` or `"printdisabled"`: replace the per-item `ia.get_metadata()` calls with **a single batched Solr query** for all needed records, using `ebook_access` (already indexed from the same IA metadata by the Solr indexer).
For call sites that fetch `external-identifier` for ACS4 DRM resource IDs: **ACS4 is deprecated** — these are dead code paths and should be removed.
---
## Call sites (book metadata only; coverstore and account metadata excluded)
### Candidates for removal/replacement
**1. `openlibrary/plugins/books/readlinks.py:229`** — Books Read API
```python
self.iaid_to_meta = {iaid: ia.get_metadata(iaid) for iaid in iaids}
```
Fetches full IA metadata for **every** IA identifier on a work (up to 500). Downstream code only uses `meta.get("collection", [])` to check `"inlibrary"` and `"printdisabled"` to determine loan status. This is the most expensive site: N separate metadata round-trips per API call.
The `ia` Solr field is already fetched for the same works via `get_solr_fields_for_works` just above this line — `ebook_access` could be fetched in the same Solr query, replacing all N metadata calls with one.
**2. `openlibrary/plugins/openlibrary/home.py:329`** — Homepage carousel
```python
collections = ia.get_metadata(d.ocaid).get("collection", [])
if "inlibrary" in collections:
d.borrow_url = ...
else:
d.read_url = ...
```
One metadata call per carousel book, only to check `"inlibrary"` in `collection`. Should be replaced with a single Solr query for `ebook_access` across all carousel items.
**3. `openlibrary/plugins/upstream/models.py:146`** — Edition pages (`Edition.get_ia_meta_fields()`)
Uses `collection` (for DAISY/printdisabled check, borrow links) and `external-identifier` (for ACS4 DRM resource IDs). The `collection`-derived logic should use `ebook_access` from Solr. The `external-identifier` / ACS4 code path should be **removed** (ACS4 is deprecated).
**4. `openlibrary/core/lending.py:959`** — `get_resource_id()`
```python
metadata = ia.get_metadata(identifier)
external_identifiers = metadata.get("external-identifier", [])
```
Fetches metadata to extract ACS4 epub/pdf resource IDs. ACS4 is deprecated — this code path should be **removed**.
### Likely necessary (for reference)
**5. `openlibrary/plugins/openlibrary/connection.py:131`** — Synthetic `/books/ia:*` pages
Constructs a fake edition record from IA metadata when no OL record exists. Full metadata needed; no alternative source.
**6. `openlibrary/plugins/importapi/code.py:273`** — Import validation
Fetches metadata to validate the item and check import eligibility. Result is passed directly to `get_marc_record_from_ia` (no double-fetch). Expected and infrequent.
**7. `openlibrary/catalog/get_ia.py:40`** — `get_marc_record_from_ia()` fallback
Only fires when called without pre-fetched `ia_metadata`. In production, `importapi` always passes metadata, so this is a defensive fallback. Low priority.
### Already optimized (for context)
**8. `openlibrary/solr/data_provider.py:106–167`** — Solr indexer
Uses `advancedsearch.php` with `doc_ids` (batch, 4 fields only: `identifier`, `boxid`, `collection`, `access-restricted-item`) and falls back to `archive.org/metadata/{ocaid}/metadata` only for bad-apple OCAIDs that break the batch. This is the right pattern and confirms `collection` / `access-restricted-item` are already indexed into OL Solr as `ebook_access`.
---
## Breakdown
Implementation Details (for maintainers)
#### Related files
- `openlibrary/core/ia.py:88–102` — `get_metadata_direct()` / `get_metadata()`
- `openlibrary/plugins/books/readlinks.py:229` — bulk metadata fetch in Read API
- `openlibrary/plugins/openlibrary/home.py:329` — carousel collection check
- `openlibrary/plugins/upstream/models.py:146` — `Edition.get_ia_meta_fields()`
- `openlibrary/core/lending.py:959` — `get_resource_id()` ACS4 lookup
- `openlibrary/solr/data_provider.py:106–167` — batch fetch + individual fallback (reference pattern)
#### Requirements Checklist
- [ ] `readlinks.py:229` — replace N `ia.get_metadata()` calls with a single Solr query for `ebook_access` across all iaids; add regression test
- [ ] `home.py:329` — replace per-item `ia.get_metadata()` with a single batched Solr `ebook_access` query across carousel items; add regression test
- [ ] `models.py:146` — replace `collection`-based checks with Solr `ebook_access`; remove `external-identifier` / ACS4 code (deprecated)
- [ ] `lending.py:959` — remove `get_resource_id()` ACS4 path (deprecated)
- [ ] Audit any remaining callers of `get_resource_id()` and remove ACS4 call chains end-to-end
- [ ] Confirm `catalog/get_ia.py:40` fallback is never reached in production (all callers pass `ia_metadata`)
#### Stakeholders
- @mekarpeles
Contributor guide
Assessment
This issue has not been assessed yet.