Feed Registry & Acquisitions Import System (Epic)

### Feature Request

# Feed Registry & Acquisitions Import System (formerly "Trusted Book Providers (TBP) Ingestion Pipeline")

## Problem / Opportunity

Open Library currently lacks a standardized mechanism for ingesting and tracking metadata from trusted external book providers such as BWB OPDS, Lenny, and Standard Ebooks.

As Open Library expands partnerships with external providers, we need infrastructure that can:

- Track trusted provider feeds and ingestion progress.
- Process provider records incrementally.
- Associate provider metadata with Open Library editions.
- Store acquisition-related metadata for downstream indexing and discovery.
- Enable future integrations with additional feed types (OPDS, ONIX, Standard Ebooks, etc.).

Without this infrastructure, onboarding new trusted providers requires ad-hoc solutions and makes ongoing synchronization difficult to maintain.

### Success Criteria

- Trusted provider feeds can be registered and tracked.
- Feed ingestion can run incrementally.
- Provider metadata is persisted against editions.
- Acquisition records are stored separately for indexing purposes.
- Solr can react to acquisition-related updates.

## Current Status (2026-07-23)

This issue was auto-closed as "completed" on 2026-06-21 when PR #12851 merged and referenced "Closes #12844" — but that PR only covered Subtask 2 of the 4 below. Reopening: this is the right issue to track the integrated end-state, since it already frames the full pipeline (registry → acquisitions → ingestion → Solr surfacing) rather than any one piece of it.

Real state of each piece, as of this update:

- **Feed Registry DB** — designed via PR #12846 (`tbp_feed_registry` table + `FeedRegistry` class), but that PR is **closed, unmerged, and now shows CONFLICTING** — needs a fresh PR rebased on current master.
- **Acquisitions DB** — table + interface **merged** via PR #12851, but not yet wired into the live import/acquisition flow (nothing populates or reads it in production yet).
- **BWB OPDS ingestion script** — drafted via PR #12852 (open, draft, mergeable), not yet integrated into the actual import flow.
- **Acquisitions in Solr** — not started. Current plan is to **dynamically fetch acquisitions from the DB at query time and weave them into results as needed**, rather than embedding acquisition data directly into the Solr index — avoids ballooning the index, but the query-time join pattern needs its own design pass (see Subtask 4 below, revised).

**Open question carried over from original triage, still unresolved:** how this relates to #5792 ("Trusted Book Providers", the longer-running epic led by @cdrini) and #12655 ("Epic: BookWorm — Modernize the Import Pipeline", a distinct-but-adjacent effort on the general import queue/DB). Worth a scope conversation before further subtask PRs land, so effort doesn't split across three epics tracking overlapping ground.

Related, not yet reconciled:
- #5792 — Trusted Book Providers (epic)
- #10251 — Simplify Trusted Book Providers integration
- #12655 — Epic: BookWorm — Modernize the Import Pipeline
- #11264 — Add Solr acquisitions field for BookServer integration

## Proposal

Introduce a Trusted Book Provider (TBP) ingestion pipeline consisting of:

1. A `tbp_feed_registry` table to track trusted feeds and ingestion progress.
2. An `acquisitions` table for provider-specific acquisition metadata.
3. A cron-based ingestion process that fetches and processes provider feeds.
4. Solr integration that recognizes acquisition updates and provider metadata.

The initial implementation will focus on:

- BWB OPDS
- Internet Archive Labs Lenny

Future providers such as Standard Ebooks and ONIX feeds should be supported through the same framework.

### Breakdown

### Parent Issue Tracking

- [ ] 1. Create `tbp_feed_registry` table and seed initial provider entries — PR #12846 (closed, unmerged, conflicting — needs a fresh PR)
- [x] 2. Create `acquisitions` table for acquisition metadata — PR #12851 (merged; not yet wired into live flow)
- [ ] 3. Implement Trusted Book Provider ingestion cron — PR #12852 (open, draft; BWB script drafted, not integrated)
- [ ] 4. Extend solr-updater to process acquisition updates — not started; revised plan is query-time DB fetch, not index-time embedding (see Current Status above)

### Subtask 1: Create `tbp_feed_registry` Table

**Current status:** Designed in PR #12846, but that PR is closed/unmerged and now conflicting. Needs a fresh PR before this can move forward.

**Goal**

Track available provider feeds and ingestion progress.

**Requirements**

- Create migration for `tbp_feed_registry`
- Store:
- provider name
- feed type (`opds`, `onix`, `standardebooks`, etc.)
- feed URL
- last processed timestamp
- last fetch date
- metadata JSON
- Manually seed:
- BWB OPDS
- Lenny

**Acceptance Criteria**

- Registry table exists.
- Feed metadata can be stored.
- Initial provider records can be added manually.

---

### Subtask 2: Create `acquisitions` Table

**Current status:** Merged via PR #12851. Table and interface exist, but nothing in the live import/acquisition flow populates or reads it yet — still needs wiring.

**Goal**

Store acquisition metadata associated with Open Library editions and works.

**Requirements**

- Create migration for `acquisitions`
- Store:
- work key
- edition key
- provider
- acquisition timestamp
- metadata JSON blob
- Support future acquisition metadata such as pricing and provider-specific data.
- Ensure compatibility with work/edition merges through `resolve_redirects`.

**Acceptance Criteria**

- Acquisition records can be inserted and updated.
- Records remain valid after merges.
- Metadata blob supports future expansion.

---

### Subtask 3: Implement Trusted Book Provider Ingestion Cron

**Current status:** BWB OPDS feed importer script drafted in PR #12852 (open, draft, mergeable), not yet integrated into the actual ingestion flow.

**Goal**

Fetch provider feeds and update Open Library records.

**Requirements**

- Daily cron execution.
- Read providers from `tbp_feed_registry`.
- Initial support for:
- BWB OPDS
- Lenny
- For BWB:
- Fetch all records initially.
- Track update timestamps for future incremental processing.
- Query Open Library editions for latest provider identifiers.
- Update:
- Edition metadata (when necessary)
- Acquisition records
- Persist ingestion state back to registry.

**Acceptance Criteria**

- Cron successfully processes BWB feed.
- Edition records are updated.
- Acquisition records are upserted.
- Registry state advances after successful ingestion.

---

### Subtask 4: Extend Solr Updater for Acquisition Support

**Current status:** Not started. Revised plan (2026-07-23): rather than extending solr-updater to embed acquisition data at index time, dynamically fetch acquisitions from the `acquisitions` DB table at query time and weave them into results as needed. Avoids ballooning the Solr index, but needs care around query patterns (efficient batch lookup for a page of search results, caching) to avoid trading index bloat for query-time latency.

**Goal**

Allow acquisition and provider metadata to be surfaced in search/discovery results.

**Requirements (revised)**

- Efficient query-time lookup of acquisition records for a set of editions/works returned by a search.
- Avoid N+1 DB queries per search result page.
- Surface provider metadata (price, download options, etc.) alongside search results.

**Original requirements (index-time approach, superseded above, kept for reference)**

- Extend solr-updater to:
- Detect TBP-related edition changes.
- Detect acquisition record changes.
- Trigger reindexing when:
- Edition metadata changes.
- Acquisition metadata changes.
- Surface provider metadata within indexed records.

**Acceptance Criteria**

- Acquisition data is surfaced in search results without materially increasing Solr index size.
- Provider metadata is available for display/filtering.
- Existing indexing behavior remains unaffected.

Implementation Details (for maintainers)

#### Related files

Potential areas of implementation:

- Database migrations
- `openlibrary/core/`
- `openlibrary/plugins/`
- `scripts/`
- `openlibrary/solr/`
- Cron configuration

Refer to:
https://docs.openlibrary.org/developers/backend/endpoints.html

#### Requirements Checklist

* [ ] Create `tbp_feed_registry` table (PR #12846 needs a fresh, unblocked replacement).
* [ ] Seed BWB and Lenny feed records.
* [x] Create `acquisitions` table (PR #12851, merged).
* [ ] Wire `acquisitions` table into the live import/acquisition flow.
* [ ] Implement feed ingestion cron (PR #12852 drafts the BWB piece; not yet integrated).
* [ ] Update editions with provider metadata.
* [ ] Upsert acquisition records.
* [ ] Design + implement query-time acquisition lookup for search results (revised Subtask 4).
* [ ] Add relevant documentation.

#### Stakeholders

* @mekarpeles
* @cdrini
* @ronibhakta1

---

#### Instructions for Contributors

* **Before** creating a new branch or pushing changes to a PR, please first run the repository update commands to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.