benbjohnson / benbjohnson/litestream

VFS write buffer race condition causes 'read page from buffer: EOF' errors

Open
#1,018 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.4k
Forks
414
Avg merge
7d 1h
Merged PRs (30d)
21

Description

## Description

When using the Litestream VFS with write support enabled and a short sync interval (1 second), periodic sync errors occur:

```
ERROR periodic sync failed component=litestream-vfs vfs=true name=/var/lib/juicefs/meta.db error="upload LTX: read page 1 from buffer: EOF"
```

## Steps to Reproduce

1. Create a VFS with `WriteEnabled = true` and `WriteSyncInterval = 1 * time.Second`
2. Open a SQLite database through the VFS
3. Perform writes to the database
4. Observe periodic sync errors in the logs

## Expected Behavior

The sync loop should successfully read dirty pages from the write buffer and sync them to the remote replica.

## Actual Behavior

The sync loop attempts to read pages from the buffer file at offsets stored in the `dirty` map, but the buffer file returns EOF, indicating it's empty or shorter than expected.

## Analysis

Looking at the code in `vfs.go`, the issue appears to be a race condition:

1. `WriteAt()` calls `writeToBuffer()` which writes page data to the buffer file and updates `f.dirty[pgno]` with the buffer offset
2. The sync loop in `syncLoop()` calls `syncToRemote()` which calls `createLTXFromDirty()`
3. `createLTXFromDirty()` reads from `bufferFile.ReadAt(data, bufferOff)` using offsets from `f.dirty`

The race appears to be between:
- Writing to the buffer file and updating `f.dirty`
- The sync loop reading from the buffer file

Possible causes:
1. The buffer file write hasn't been flushed to disk yet
2. The `f.dirty` map is updated before the buffer write completes
3. File handle buffering issues

## Workaround

Using a longer sync interval (10 seconds) avoids the issue, suggesting it's timing-related.

## Environment

- Litestream version: v0.5.6
- Go version: 1.24
- OS: Linux (Docker container)
- Architecture: arm64

## Context

Discovered while integrating Litestream VFS with JuiceFS for on-demand SQLite metadata fetching.

Contributor guide

Open the contributing guide

Research direction

Start in vfs.go by tracing writeToBuffer and WriteAt alongside syncLoop, syncToRemote, and createLTXFromDirty. Reproduce the failure with WriteEnabled and a one-second WriteSyncInterval, then verify that dirty pages remain readable during sync and the reported EOF errors no longer occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sqlite
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.