internetarchive / internetarchive/openlibrary

Amazon 979-prefix ISBN-13 lookup via Creators API search_items

Open
#13,316 2 comments 0 reactions 1 assignee Claimed by @openlibrary-bot View on GitHub
Lead: @mekarpeles Module: Import Needs: Review Assignee Priority: 3 Theme: Affiliate API Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

## Summary

Books with a **979-prefix ISBN-13 can never be looked up on Amazon**. They are silently rejected by the affiliate server before any Amazon call is made, so we get no price, no affiliate link, and no Amazon import metadata for them — even though Amazon stocks these books.

The fix is to resolve them via the Creators API `search_items` operation, which `AmazonCreatorsAPI` does not currently expose.

## Why 979 ISBNs are structurally excluded

Amazon's `get_items` is an **exact ID lookup**: it accepts an ISBN-10 or a real ASIN, nothing else. A 979-prefix ISBN-13 has **no ISBN-10 equivalent** (the 979 range was allocated precisely because the 978 space ran out, and there is no back-conversion). Amazon assigns such books an arbitrary `B*` ASIN with no algorithmic relationship to the ISBN, so no amount of conversion will find it. Only a **keyword search** can.

This is the same root fact already documented in `amazon_affiliate_url()` (from #6572, per @hornc):

https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/vendors.py#L311-L336

That fix handled the *outbound link* case by falling back to an Amazon search URL. The *metadata lookup* case was never addressed.

## Where it dies today

`Submit.GET` in `scripts/affiliate_server.py` derives its Amazon key and finds nothing usable:

```python
b_asin, isbn_10, isbn_13 = normalize_identifier(identifier)
key = isbn_10 or b_asin
```

For `9791234567896` → `normalize_identifier` returns `(None, None, "9791234567896")`, so `key` is `None`. Two exits follow, neither of which touches Amazon:

- high_priority + stage_import → straight to Google Books (`stage_from_google_books`)
- otherwise → `{"error": "rejected_isbn"}`

`openlibrary/core/vendors.py::_get_amazon_metadata` gates on the 978 prefix explicitly for the same reason:

```python
if len(id_) == 13 and id_.startswith("978"):
isbn = isbn_13_to_isbn_10(id_)
```

So a 979 ISBN is never converted, never queued, and never fetched. The Amazon branch is dead code for this entire ISBN range.

## Why this is newly fixable

`AmazonCreatorsApi` exposes `search_items(keywords=..., search_index=..., item_count=...)`, returning a `SearchResult` whose `.items` are the **same `Item` type** `get_items` returns — so the existing `AmazonCreatorsAPI.serialize()` works on search results unchanged.

Our wrapper `AmazonCreatorsAPI` has **no search method at all**. Legacy `AmazonAPI.search()` exists but is explicitly CLI-only (*"Adding method to test amz searches from the CLI, unused otherwise"*) and is being removed along with PA-API in #13315.

Two details of the existing code mean the cache/import layers need **no changes**:

1. `AmazonCreatorsAPI.serialize()` already sources `isbn_13` from `external_ids.eans` rather than deriving it from the ASIN, so a `B*`-ASIN product still serializes with its true 979 ISBN-13.
2. `make_cache_key()` prefers `isbn_13` first, so the product caches under `amazon_product_9791234567896` — exactly the key `Submit.GET` already reads back.

## Proposed change

1. Add `AmazonCreatorsAPI.search_items()` — mirroring `get_products()`'s throttle discipline and its swallow-and-log-on-error contract.
2. Resolve unmatched ISBN-13s through it **only** when there is no ISBN-10 and no `B*` ASIN — i.e. exactly the path that returns `rejected_isbn` today. Strictly additive: nothing that currently works can regress.
3. **Verify before accepting.** A keyword search is not an exact-match lookup, so the returned item's `external_ids.eans` must be confirmed to contain the requested ISBN-13 before the product is cached or staged. Without this we would import the wrong book's metadata under the right ISBN. This is the most important safety property of the change.

## Constraints (#13277, #13296)

Amazon affiliate lookups caused a ~12h site-wide outage, so latency on this path is the primary risk and is treated as a hard design constraint:

- The resolution must happen in the **background `amazon_lookup` thread** (`process_amazon_batch`), never inline in `Submit.GET`. No new blocking Amazon call may be added to a web-worker request path.
- `get_items` batches up to 10 identifiers per call; **`search_items` is one ISBN per call**. A burst of 979 ISBNs therefore costs one API call each and could starve the batched path. The existing `API_MAX_ITEMS_PER_CALL` / `API_MAX_WAIT_SECONDS` budget must be respected, and the 979 partition bounded rather than unbounded.

## Not yet verified

Live Amazon behavior is **unconfirmed** — I have no Creators API credentials in this environment. Specifically unverified:

- that `search_items(keywords="<979 isbn>", search_index="Books")` reliably returns the correct edition
- whether Amazon indexes the raw ISBN-13 string at all for these items
- the real hit rate

If searching by ISBN-13 turns out not to be indexed reliably, the correct outcome is to close this rather than ship a low-hit-rate extra API call on an outage-sensitive path. **A prod spot-check should gate implementation.**

## Success criteria

- `AmazonCreatorsAPI.search_items()` exists, is throttled, and fails soft like `get_products()`
- A 979 ISBN-13 resolves to the correct Amazon product, verified via `eans` before use
- A search result whose `eans` does not contain the requested ISBN is **rejected**, not cached
- No new Amazon call on any synchronous request path
- No behavior change for 978 ISBNs or `B*` ASINs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.