bunkerity / bunkerity/bunkerweb
Deadlock in acquire_db_lock(): the wait deadline is computed before the loop
- Dominant language
- Python
- Stars
- 10.9k
- Forks
- 643
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 42
Description
### Description
`acquire_db_lock()` in `src/common/core/backup/backup.py` computes its deadline **before** the wait loop, so nothing in the loop condition depends on time passing:
```python
def acquire_db_lock():
"""Acquire the database lock to prevent concurrent access to the database."""
current_time = datetime.now().astimezone() # frozen here
while DB_LOCK_FILE.is_file() and DB_LOCK_FILE.stat().st_ctime + 30 > current_time.timestamp():
LOGGER.warning("Database is locked, waiting for it to be unlocked (timeout: 30s) ...")
sleep(1)
DB_LOCK_FILE.unlink(missing_ok=True)
DB_LOCK_FILE.touch()
```
Both operands are constant: the lock file's `st_ctime` does not change while it sits there, and `current_time` is a snapshot. So the condition can never become false on its own — if it is true on the first iteration it stays true, and the caller spins at one log line per second until the file is removed by someone else. The advertised `timeout: 30s` never fires.
Any lock younger than 30 seconds at the moment the caller arrives triggers this, not just one at the 29–30 s boundary.
The same pattern is duplicated in `src/scheduler/main.py` (the wait before `get_metadata()` in the main loop), with the same frozen snapshot. So a stuck holder parks the scheduler loop too, which is where the user-visible damage comes from.
### What it looks like in production
Observed on 1.6.11, Docker integration, MariaDB backend, single instance.
- Lock file created at 05:11:20.
- `backup-data` started at 05:11:50 and entered the loop.
- 18905 identical `Database is locked, waiting for it to be unlocked (timeout: 30s) ...` lines from the same PID over the next 5h16, while the lock file's `ctime` never moved from `05:11:20.282424958`.
- `/var/tmp/bunkerweb/scheduler.healthy` was never written, so the container sat `unhealthy` for the whole time.
- No configuration push, no certificate renewal, and the daily database backup never ran — the backups directory jumps straight from `05:08` the previous day to `10:28` on the day of the incident.
- `rm /var/lib/bunkerweb/db.lock` ended it: the backup completed in 6 seconds and the scheduler resumed its interrupted startup sequence.
Nothing else was wrong with the deployment: BunkerWeb itself stayed healthy and kept serving all vhosts with its last-pushed configuration the entire time. That is what makes this nasty — the only signal is the container healthcheck going red.
### Affected versions
Present in 1.6.11, 1.6.14 and 1.6.15-rc1 (checked against the tagged sources). Confirmed on Discord by @TheophileDiot, who reports the fix is landing in 1.6.15~rc2; opening this so it has a number to close.
### Suggested fix
Re-evaluate the deadline inside the loop, in both `acquire_db_lock()` and the `scheduler/main.py` copy, so the 30 second timeout is a real deadline rather than a frozen snapshot.
### Reproduction
```python
from datetime import datetime
from pathlib import Path
from time import sleep
DB_LOCK_FILE = Path("/tmp/db.lock")
DB_LOCK_FILE.touch() # a lock less than 30s old
current_time = datetime.now().astimezone()
while DB_LOCK_FILE.is_file() and DB_LOCK_FILE.stat().st_ctime + 30 > current_time.timestamp():
print("waiting ...") # never stops
sleep(1)
```
Contributor guide
Assessment
This issue has not been assessed yet.