dolthub / dolthub/dolt

CLI writes fail with 'database is read only' due to stale journal lock after process exit

Open
#10,852 6 comments 0 reactions 0 assignees View on GitHub
bad error message bug cli concurrency customer issue
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 5h
Merged PRs (30d)
108

Description

## Summary

After a `dolt sql` process exits (or is killed), subsequent CLI write operations fail with `cannot update manifest: database is read only`, even though:
- The process is gone (`ps` confirms no PID)
- `lsof` shows no process holding the journal file
- File permissions are correct (`-rw-------`)
- `dolt fsck` reports no problems
- Read operations (`dolt sql -q "SELECT ..."`) work fine

## Environment
- Dolt: 1.86.1
- OS: Arch Linux 6.19.6 (ext4, local disk)
- No `dolt sql-server` running
- No `.dolt/sql-server.info` file

## Reproduction

1. Run a long batch of SQL operations:
```bash
dolt sql < large_batch.sql # 500+ INSERT/DELETE/UPDATE statements
```
2. Run concurrent read-only queries from another process during step 1
3. After step 1 completes, try any write:
```bash
dolt add . # fails: "cannot update manifest: database is read only"
dolt sql -q "DELETE FROM t WHERE ..." # also fails
dolt status # works fine
dolt sql -q "SELECT ..." # works fine
```

## Analysis

The journal file `.dolt/noms/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv` appears to retain a stale `fslock` after the process that held it exits. `fuser` initially shows the PID, but `ps` shows the process no longer exists. The lock is never released, causing all subsequent write paths to silently fall back to read-only mode.

This is related to the `FailOnLockTimeout` behavior in `go/store/nbs/journal.go` — when a CLI command can't acquire the journal lock within 100ms, it opens read-only instead of erroring. But in this case, the lock holder is dead and the lock should have been released by the OS.

## Expected behavior

1. After a dolt process exits, the journal lock should be released
2. If a stale lock is detected (holder PID doesn't exist), dolt should clean it up
3. CLI commands should fail with a clear error ("database locked by PID X") rather than silently opening read-only

## Workaround

Currently the only workaround is to:
1. Wait some time (sometimes works)
2. Or restart the shell / re-clone the database

## Related
- #9312 (similar error with sql-server)
- PR #10838 (fixed race in GC PruneTableFiles, but this is a different lock issue)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.