internetarchive / internetarchive/openlibrary
Why does my Local Dev Instance have no books?
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
## Problem
A developer's local instance showed no books/documents — Solr appeared empty. The root cause was that the **Docker base image had updated** (e.g. the Python 3.14 bump landed in master) **without the developer having rebased their branch onto updated master first**. This caused a mismatch between the running Solr container and what the codebase expected, leaving Solr in a broken state where nothing gets indexed.
### Symptoms
- Search returns no results
- Book/work pages load but the site feels empty
- `solr-updater` may log errors or silently fail to index
### Root cause confirmed by contributor:
> "The issue was Solr. More specifically, it was getting messed up by the switch to Python 3.14, due to the image having updated without me having properly rebased onto the updated master branch. After fixing that and doing `docker volume rm ` the problem has been resolved."
---
## Why existing safeguards didn't catch it
- **PR #11902** added a Solr auto-rebuild health check that detects schema divergence at startup — but this scenario (base image version change without a code rebase) isn't caught by schema comparison alone.
- **Issue #12792** is tracking Solr _version_ mismatch detection, which is the underlying gap.
- **`scripts/fully_reset_environment.sh`** (added in PR #12318) and the [Solr wiki](https://github.com/internetarchive/openlibrary/wiki/Solr) both document volume removal, but neither specifically calls out the "image updated before rebasing" trigger.
---
## Debug steps (proposed)
When local dev shows no books, start by isolating whether the problem is the **database (infobase/PostgreSQL)** or **Solr**. They are independent layers and fail differently.
### Step 1 — Is it the DB or Solr?
**Check the DB directly** via the `query.json` API (bypasses Solr entirely):
```
http://localhost:8080/query.json?type=/type/work&limit=1
```
Or via curl:
```bash
curl "http://localhost:8080/query.json?type=/type/work&limit=1"
```
| Result | Meaning |
|--------|---------|
| Returns a work record | DB is fine → problem is Solr |
| Returns `[]` or an error | DB itself is empty or broken |
You can also check a known key directly:
```bash
curl "http://localhost:8080/works/OL45804W.json"
```
If that 404s, the DB has no data. If it returns JSON, the DB is fine and Solr is the culprit.
### Step 2 — If it's Solr
**Verify Solr document count:**
```bash
curl "http://localhost:8983/solr/openlibrary/select?q=*:*&rows=0" | python3 -m json.tool | grep numFound
```
`numFound: 0` confirms Solr is empty.
**Check solr-updater logs for errors:**
```bash
docker compose logs solr-updater --tail=50
```
**Check whether you're up to date with master** (a common trigger — base image updates like a Python version bump can break Solr volumes if you haven't rebased):
```bash
git fetch origin
git log --oneline HEAD..origin/master
```
**Fix: rebase, then remove stale Solr volumes:**
```bash
git fetch origin && git rebase origin/master
docker compose build
docker compose stop solr solr-updater
docker volume rm openlibrary_solr-data openlibrary_solr-updater-data
docker compose up
```
Or use the full reset script:
```bash
bash scripts/fully_reset_environment.sh
```
### Step 3 — If it's the DB
**Check PostgreSQL directly:**
```bash
docker compose exec db psql -U openlibrary openlibrary -c "SELECT COUNT(*) FROM thing;"
```
If count is 0, the DB was never seeded or the volume was wiped.
**Check infobase/home logs:**
```bash
docker compose logs home --tail=50
docker compose logs db --tail=30
```
**Check DB container health:**
```bash
docker compose ps db
```
**Fix: reload test data.** The dev environment seeds from `infogami/test-database.tar.gz`. If the DB volume is empty or corrupt:
```bash
# Full reset (nukes all volumes including DB):
bash scripts/fully_reset_environment.sh
# Or selectively remove just the DB volume:
docker compose stop home db
docker volume rm openlibrary_db-data
docker compose up
```
Note: after a DB volume reset, Solr will also need a reindex since its data references DB records:
```bash
docker compose run --rm home make reindex-solr
```
---
## Proposed fix
Add a "No books showing / site feels empty" section to:
- [`docs/developers/misc/common-issues.md`](https://docs.openlibrary.org/developers/misc/common-issues.html) — currently only covers 404s and finding endpoints
- The [Solr wiki page](https://github.com/internetarchive/openlibrary/wiki/Solr) — under a "Solr is empty / no documents" heading
- Optionally: `docker/README.md` "Been away a while?" section — explicitly call out that base image updates (Python bumps, etc.) require a rebase + volume reset
Longer-term: the auto-rebuild health check from PR #11902 could be extended to also detect base image version mismatches (tracked in #12792).
---
## Related issues / PRs
- #11013 — original trending updater / startup race condition
- #11012 — missing docker volumes in reset command (docs fix)
- #11902 — Solr auto-rebuild health check for local dev (schema divergence)
- #12318 — `fully_reset_environment.sh` script
- #12325 — home waits for Solr via healthcheck
- #12786 — Update to Solr 10 (schema changes; likely recent trigger)
- #12792 — Detect Solr version mismatch at container start (open, related)
Contributor guide
Assessment
This issue has not been assessed yet.