benbjohnson / benbjohnson/litestream
VFS write buffer race condition causes 'read page from buffer: EOF' errors
- Dominant language
- Go
- Stars
- 14.4k
- Forks
- 414
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 21
Description
## Description
When using the Litestream VFS with write support enabled and a short sync interval (1 second), periodic sync errors occur:
```
ERROR periodic sync failed component=litestream-vfs vfs=true name=/var/lib/juicefs/meta.db error="upload LTX: read page 1 from buffer: EOF"
```
## Steps to Reproduce
1. Create a VFS with `WriteEnabled = true` and `WriteSyncInterval = 1 * time.Second`
2. Open a SQLite database through the VFS
3. Perform writes to the database
4. Observe periodic sync errors in the logs
## Expected Behavior
The sync loop should successfully read dirty pages from the write buffer and sync them to the remote replica.
## Actual Behavior
The sync loop attempts to read pages from the buffer file at offsets stored in the `dirty` map, but the buffer file returns EOF, indicating it's empty or shorter than expected.
## Analysis
Looking at the code in `vfs.go`, the issue appears to be a race condition:
1. `WriteAt()` calls `writeToBuffer()` which writes page data to the buffer file and updates `f.dirty[pgno]` with the buffer offset
2. The sync loop in `syncLoop()` calls `syncToRemote()` which calls `createLTXFromDirty()`
3. `createLTXFromDirty()` reads from `bufferFile.ReadAt(data, bufferOff)` using offsets from `f.dirty`
The race appears to be between:
- Writing to the buffer file and updating `f.dirty`
- The sync loop reading from the buffer file
Possible causes:
1. The buffer file write hasn't been flushed to disk yet
2. The `f.dirty` map is updated before the buffer write completes
3. File handle buffering issues
## Workaround
Using a longer sync interval (10 seconds) avoids the issue, suggesting it's timing-related.
## Environment
- Litestream version: v0.5.6
- Go version: 1.24
- OS: Linux (Docker container)
- Architecture: arm64
## Context
Discovered while integrating Litestream VFS with JuiceFS for on-demand SQLite metadata fetching.
Contributor guide
Research direction
Start in vfs.go by tracing writeToBuffer and WriteAt alongside syncLoop, syncToRemote, and createLTXFromDirty. Reproduce the failure with WriteEnabled and a one-second WriteSyncInterval, then verify that dirty pages remain readable during sync and the reported EOF errors no longer occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100