internetarchive / internetarchive/openlibrary
Linux Docker Issues: Fresh clone does not boot: web exits 3 writing _testing-prs.json into a bind mount it cannot write
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Problem
A fresh clone doesn't boot. `docker compose up -d` brings every service up, then `web` exits with code 3 and nothing serves on :8080.
```
File "/openlibrary/openlibrary/plugins/openlibrary/status.py", line 695, in _ensure_testing_state_file
TESTING_STATE_FILE.write_text(json.dumps({"last_deploy_at": "", "prs": []}, indent=2))
PermissionError: [Errno 13] Permission denied: '_testing-prs.json'
[ERROR] Worker (pid:9) exited with code 3
[ERROR] Shutting down: Master
```
`_ensure_testing_state_file()` runs at import time via `setup()` and writes `_testing-prs.json` into the repo root. The repo root is the bind mount, owned by the host user, while the container runs as `openlibrary` uid 999 — so the write fails and gunicorn's worker never boots.
`_testing-prs.json` is gitignored (`.gitignore:75`), so it is never present in a fresh clone and the `if not TESTING_STATE_FILE.exists()` guard always fires.
This looks like it contradicts the rule stated in `docker/Dockerfile.olbase` itself: *"We use 999:999 for the openlibrary user. Any volume mounts which require read/write access by the container should be set to this user."* The repo bind mount isn't, and the app now writes to it on startup.
I suspect this is invisible on macOS and Docker Desktop, where the file sharing layer maps ownership so the write succeeds. It reproduces reliably on native Docker under Linux/WSL2, where the host user is typically uid 1000.
### Reproducing the bug
1. Clone master fresh (I was on `e5b6d22c8`) and run `make git`
2. `docker compose up -d` on native Docker (not Docker Desktop), host user uid 1000
3. `curl http://localhost:8080/`
* Expected behavior: the site serves
* Actual behavior: `web` exited (3), connection refused; `docker compose logs web` shows the traceback above
Deterministic in both directions: `rm _testing-prs.json && docker compose restart web` fails every time, and recreating the file and restarting brings it straight back up.
### Context
- Browser (Chrome, Safari, Firefox, etc): n/a
- OS (Windows, Mac, etc): Windows 11 + WSL2, Ubuntu 26.04
- Logged in (Y/N): n/a
- Environment (prod, dev, local): local — Docker Engine 29.1.3, Compose 2.40.3, no Docker Desktop
### Breakdown
Implementation Details (for maintainers)
The file backs the `/status` page and the testing-status API rather than anything core, so tolerating an unwritable path and letting that page degrade seems proportionate. Writing to a container-writable location instead would also fix it. I didn't want to assume which you'd prefer.
Workaround for anyone blocked meanwhile: create `_testing-prs.json` in the repo root before first boot.
#### Requirements Checklist
* [ ] `docker compose up -d` on a fresh clone serves :8080 with no manually created files
* [ ] startup no longer fails when the repo root isn't writable by uid 999
#### Related files
* `openlibrary/plugins/openlibrary/status.py:695` — `_ensure_testing_state_file()`, added in #13278
* `docker/Dockerfile.olbase:22-23` — pins the container user to uid 999
* `.gitignore:75` — `/_testing-prs.json`
#### Stakeholders
* @RayBB (#13278)
#### Instructions for Contributors
- Please [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Research direction
Start with openlibrary/plugins/openlibrary/status.py:695 and trace _ensure_testing_state_file() through setup(), then review docker/Dockerfile.olbase:22-23 and the ignored path at .gitignore:75. Reproduce with a fresh clone and docker compose up -d on native Linux Docker. Done means the web service boots and serves :8080 without manually creating _testing-prs.json, even when the repository bind mount is not writable by uid 999.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, python
- Domain
- backend, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100