CLI writes fail with 'database is read only' due to stale journal lock after process exit
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 108
Description
## Summary
After a `dolt sql` process exits (or is killed), subsequent CLI write operations fail with `cannot update manifest: database is read only`, even though:
- The process is gone (`ps` confirms no PID)
- `lsof` shows no process holding the journal file
- File permissions are correct (`-rw-------`)
- `dolt fsck` reports no problems
- Read operations (`dolt sql -q "SELECT ..."`) work fine
## Environment
- Dolt: 1.86.1
- OS: Arch Linux 6.19.6 (ext4, local disk)
- No `dolt sql-server` running
- No `.dolt/sql-server.info` file
## Reproduction
1. Run a long batch of SQL operations:
```bash
dolt sql < large_batch.sql # 500+ INSERT/DELETE/UPDATE statements
```
2. Run concurrent read-only queries from another process during step 1
3. After step 1 completes, try any write:
```bash
dolt add . # fails: "cannot update manifest: database is read only"
dolt sql -q "DELETE FROM t WHERE ..." # also fails
dolt status # works fine
dolt sql -q "SELECT ..." # works fine
```
## Analysis
The journal file `.dolt/noms/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv` appears to retain a stale `fslock` after the process that held it exits. `fuser` initially shows the PID, but `ps` shows the process no longer exists. The lock is never released, causing all subsequent write paths to silently fall back to read-only mode.
This is related to the `FailOnLockTimeout` behavior in `go/store/nbs/journal.go` — when a CLI command can't acquire the journal lock within 100ms, it opens read-only instead of erroring. But in this case, the lock holder is dead and the lock should have been released by the OS.
## Expected behavior
1. After a dolt process exits, the journal lock should be released
2. If a stale lock is detected (holder PID doesn't exist), dolt should clean it up
3. CLI commands should fail with a clear error ("database locked by PID X") rather than silently opening read-only
## Workaround
Currently the only workaround is to:
1. Wait some time (sometimes works)
2. Or restart the shell / re-clone the database
## Related
- #9312 (similar error with sql-server)
- PR #10838 (fixed race in GC PruneTableFiles, but this is a different lock issue)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.