internetarchive / internetarchive/openlibrary

Linux Docker Issues: Fresh clone does not boot: web exits 3 writing _testing-prs.json into a bind mount it cannot write

Open
#13,586 6 comments 0 reactions 0 assignees View on GitHub
Affects: Configuration Affects: Server Lead: @RayBB Module: Docker Needs: Response Needs: Staff Decision Priority: 3
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

A fresh clone doesn't boot. `docker compose up -d` brings every service up, then `web` exits with code 3 and nothing serves on :8080.

```
File "/openlibrary/openlibrary/plugins/openlibrary/status.py", line 695, in _ensure_testing_state_file
TESTING_STATE_FILE.write_text(json.dumps({"last_deploy_at": "", "prs": []}, indent=2))
PermissionError: [Errno 13] Permission denied: '_testing-prs.json'
[ERROR] Worker (pid:9) exited with code 3
[ERROR] Shutting down: Master
```

`_ensure_testing_state_file()` runs at import time via `setup()` and writes `_testing-prs.json` into the repo root. The repo root is the bind mount, owned by the host user, while the container runs as `openlibrary` uid 999 — so the write fails and gunicorn's worker never boots.

`_testing-prs.json` is gitignored (`.gitignore:75`), so it is never present in a fresh clone and the `if not TESTING_STATE_FILE.exists()` guard always fires.

This looks like it contradicts the rule stated in `docker/Dockerfile.olbase` itself: *"We use 999:999 for the openlibrary user. Any volume mounts which require read/write access by the container should be set to this user."* The repo bind mount isn't, and the app now writes to it on startup.

I suspect this is invisible on macOS and Docker Desktop, where the file sharing layer maps ownership so the write succeeds. It reproduces reliably on native Docker under Linux/WSL2, where the host user is typically uid 1000.

### Reproducing the bug

1. Clone master fresh (I was on `e5b6d22c8`) and run `make git`
2. `docker compose up -d` on native Docker (not Docker Desktop), host user uid 1000
3. `curl http://localhost:8080/`

* Expected behavior: the site serves
* Actual behavior: `web` exited (3), connection refused; `docker compose logs web` shows the traceback above

Deterministic in both directions: `rm _testing-prs.json && docker compose restart web` fails every time, and recreating the file and restarting brings it straight back up.

### Context

- Browser (Chrome, Safari, Firefox, etc): n/a
- OS (Windows, Mac, etc): Windows 11 + WSL2, Ubuntu 26.04
- Logged in (Y/N): n/a
- Environment (prod, dev, local): local — Docker Engine 29.1.3, Compose 2.40.3, no Docker Desktop

### Breakdown

Implementation Details (for maintainers)

The file backs the `/status` page and the testing-status API rather than anything core, so tolerating an unwritable path and letting that page degrade seems proportionate. Writing to a container-writable location instead would also fix it. I didn't want to assume which you'd prefer.

Workaround for anyone blocked meanwhile: create `_testing-prs.json` in the repo root before first boot.

#### Requirements Checklist
* [ ] `docker compose up -d` on a fresh clone serves :8080 with no manually created files
* [ ] startup no longer fails when the repo root isn't writable by uid 999

#### Related files

* `openlibrary/plugins/openlibrary/status.py:695` — `_ensure_testing_state_file()`, added in #13278
* `docker/Dockerfile.olbase:22-23` — pins the container user to uid 999
* `.gitignore:75` — `/_testing-prs.json`

#### Stakeholders

* @RayBB (#13278)


#### Instructions for Contributors

- Please [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start with openlibrary/plugins/openlibrary/status.py:695 and trace _ensure_testing_state_file() through setup(), then review docker/Dockerfile.olbase:22-23 and the ignored path at .gitignore:75. Reproduce with a fresh clone and docker compose up -d on native Linux Docker. Done means the web service boots and serves :8080 without manually creating _testing-prs.json, even when the repository bind mount is not writable by uid 999.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, python
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.