internetarchive / internetarchive/openlibrary

[Performance] Audit and reduce direct calls to archive.org/metadata/{ocaid}

Open
#12,431 5 comments 0 reactions 1 assignee Claimed by @mekarpeles View on GitHub
Lead: @mekarpeles metadata Module: Borrowing / Lending Module: Solr Needs: Staff / Internal Priority: 2 Theme: Performance Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

## Problem / Opportunity

Open Library calls `https://archive.org/metadata/{ocaid}` (via `ia.get_metadata()` or direct HTTP) in several places — some of which are hot paths and fetch the full item metadata payload when only a single field (`collection`) is actually used. These are expensive round-trips to archive.org, and in several cases the needed data is already available in OL's own Solr index (e.g. `ebook_access`, derived during indexing from the same IA metadata).

The goal is to identify which calls are necessary vs. redundant, and eliminate or replace those that can use local Solr data instead.

**Success criteria:** Reduce the number of cold-path `archive.org/metadata/{ocaid}` calls on page renders and the Books Read API without regression to availability display or borrowing behavior.

## Proposal

For call sites that only use `collection` to check `"inlibrary"` or `"printdisabled"`: replace the per-item `ia.get_metadata()` calls with **a single batched Solr query** for all needed records, using `ebook_access` (already indexed from the same IA metadata by the Solr indexer).

For call sites that fetch `external-identifier` for ACS4 DRM resource IDs: **ACS4 is deprecated** — these are dead code paths and should be removed.

---

## Call sites (book metadata only; coverstore and account metadata excluded)

### Candidates for removal/replacement

**1. `openlibrary/plugins/books/readlinks.py:229`** — Books Read API

```python
self.iaid_to_meta = {iaid: ia.get_metadata(iaid) for iaid in iaids}
```

Fetches full IA metadata for **every** IA identifier on a work (up to 500). Downstream code only uses `meta.get("collection", [])` to check `"inlibrary"` and `"printdisabled"` to determine loan status. This is the most expensive site: N separate metadata round-trips per API call.

The `ia` Solr field is already fetched for the same works via `get_solr_fields_for_works` just above this line — `ebook_access` could be fetched in the same Solr query, replacing all N metadata calls with one.

**2. `openlibrary/plugins/openlibrary/home.py:329`** — Homepage carousel

```python
collections = ia.get_metadata(d.ocaid).get("collection", [])
if "inlibrary" in collections:
d.borrow_url = ...
else:
d.read_url = ...
```

One metadata call per carousel book, only to check `"inlibrary"` in `collection`. Should be replaced with a single Solr query for `ebook_access` across all carousel items.

**3. `openlibrary/plugins/upstream/models.py:146`** — Edition pages (`Edition.get_ia_meta_fields()`)

Uses `collection` (for DAISY/printdisabled check, borrow links) and `external-identifier` (for ACS4 DRM resource IDs). The `collection`-derived logic should use `ebook_access` from Solr. The `external-identifier` / ACS4 code path should be **removed** (ACS4 is deprecated).

**4. `openlibrary/core/lending.py:959`** — `get_resource_id()`

```python
metadata = ia.get_metadata(identifier)
external_identifiers = metadata.get("external-identifier", [])
```

Fetches metadata to extract ACS4 epub/pdf resource IDs. ACS4 is deprecated — this code path should be **removed**.

### Likely necessary (for reference)

**5. `openlibrary/plugins/openlibrary/connection.py:131`** — Synthetic `/books/ia:*` pages
Constructs a fake edition record from IA metadata when no OL record exists. Full metadata needed; no alternative source.

**6. `openlibrary/plugins/importapi/code.py:273`** — Import validation
Fetches metadata to validate the item and check import eligibility. Result is passed directly to `get_marc_record_from_ia` (no double-fetch). Expected and infrequent.

**7. `openlibrary/catalog/get_ia.py:40`** — `get_marc_record_from_ia()` fallback
Only fires when called without pre-fetched `ia_metadata`. In production, `importapi` always passes metadata, so this is a defensive fallback. Low priority.

### Already optimized (for context)

**8. `openlibrary/solr/data_provider.py:106–167`** — Solr indexer
Uses `advancedsearch.php` with `doc_ids` (batch, 4 fields only: `identifier`, `boxid`, `collection`, `access-restricted-item`) and falls back to `archive.org/metadata/{ocaid}/metadata` only for bad-apple OCAIDs that break the batch. This is the right pattern and confirms `collection` / `access-restricted-item` are already indexed into OL Solr as `ebook_access`.

---

## Breakdown

Implementation Details (for maintainers)

#### Related files

- `openlibrary/core/ia.py:88–102` — `get_metadata_direct()` / `get_metadata()`
- `openlibrary/plugins/books/readlinks.py:229` — bulk metadata fetch in Read API
- `openlibrary/plugins/openlibrary/home.py:329` — carousel collection check
- `openlibrary/plugins/upstream/models.py:146` — `Edition.get_ia_meta_fields()`
- `openlibrary/core/lending.py:959` — `get_resource_id()` ACS4 lookup
- `openlibrary/solr/data_provider.py:106–167` — batch fetch + individual fallback (reference pattern)

#### Requirements Checklist

- [ ] `readlinks.py:229` — replace N `ia.get_metadata()` calls with a single Solr query for `ebook_access` across all iaids; add regression test
- [ ] `home.py:329` — replace per-item `ia.get_metadata()` with a single batched Solr `ebook_access` query across carousel items; add regression test
- [ ] `models.py:146` — replace `collection`-based checks with Solr `ebook_access`; remove `external-identifier` / ACS4 code (deprecated)
- [ ] `lending.py:959` — remove `get_resource_id()` ACS4 path (deprecated)
- [ ] Audit any remaining callers of `get_resource_id()` and remove ACS4 call chains end-to-end
- [ ] Confirm `catalog/get_ia.py:40` fallback is never reached in production (all callers pass `ia_metadata`)

#### Stakeholders

- @mekarpeles

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.