internetarchive / internetarchive/openlibrary
perf: N+1 queries on thing table causing 3.8M single-row DB lookups
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Feature Request
## Database performance: N+1 queries on `thing` table causing 3.8M single-row lookups
### Summary
`pg_stat_statements` profiling on production reveals `SELECT * FROM thing WHERE key=$1` is the highest call-volume query on the database with **3.8 million calls** in the observation window, totalling ~2.5M ms of DB time. The batched equivalent (`SELECT ... WHERE key IN (...)`) exists and is used in some paths but only accounts for 134k calls — a 28:1 ratio of unbatched to batched lookups.
### Evidence
From `pg_stat_statements` on production (ol-db0):
```
query | calls | mean_exec_time | total_exec_time
----------------------------------------+---------+----------------+----------------
SELECT * FROM thing WHERE key=$1 | 3830328 | 0.66ms | 2,537,916ms
SELECT thing.key FROM thing ... IN ($1) | 134057 | 11.44ms | 1,533,958ms
```
The individual lookups are fast (0.66ms mean) but the volume is the problem. The server has 15GB RAM against a 892GB database, so cache pressure is real — stddev of 17ms on a 0.66ms mean indicates frequent cache misses costing 20-30ms per cold lookup.
### Expected behaviour
Things should be loaded in batches using the existing `WHERE key IN (...)` path wherever possible, reducing round-trips and I/O pressure.
### Steps to reproduce
Any page that loads a list of works/editions/authors and then resolves each one individually — likely in `openlibrary/plugins/worksearch`, `openlibrary/plugins/books`, or any template that iterates and calls `web.ctx.site.get()` in a loop.
### Suggested fix
Audit call sites of `web.ctx.site.get(key)` (single-key) and replace loops with `web.ctx.site.get_many(keys)` (batched) where the keys are known upfront. The infobase layer already supports bulk fetching.
### Impact
- Directly reduces DB query volume and I/O pressure
- No infrastructure changes required
- Likely the highest ROI performance fix available given current hardware constraints
### Environment
- PostgreSQL 18.3
- Database size: 892GB
- Server RAM: 15GB
- `shared_buffers`: 4GB
### Breakdown
Implementation Details (for maintainers)
#### Related files
Refer to [this map of common Endpoints](https://docs.openlibrary.org/developers/backend/endpoints.html):
*
#### Requirements Checklist
Checklist of requirements that need to be satisfied in order for this issue to be closed:
* [ ]
#### Stakeholders
*
#### Instructions for Contributors
* **Before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) or pushing up changes to a PR, please first [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date, as the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Assessment
This issue has not been assessed yet.