facebook / facebook/rocksdb

[BUG] RocksDB hole recovery after a crash when deployed on GlusterFS

Open
#11,149 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

We run a simple workload that inserts three key-value pairs to RocksDB under non-sync mode with WAL enabled and `manual_wal_flush` disabled. The first two inserts reach to first WAL file `4.log`, while the third insert reaches to a different WAL file `8.log`.

The simplified file system trace we observed is shown below,

```py
# Append on 4.log
1 creat("4.log")
2 write("4.log") # first insert
3 write("4.log") # second insert (may get reordered)
# Append on 8.log
4 creat("8.log")
5 write("8.log") # third insert

# [1] A crash here can lead to hole recovery problem

6 Log compaction process in the middle

# [2] Line 3 will be reordered here to persist

# Remove 4.log as the data is compacted to sst file
7 close("4.log")
8 unlink("4.log")
```

When deployed on GlusterFS, the write for second insert (line 3) may return with successful code immediately, but the data is buffered and delayed to submit to file system server to persist, which is due to a common approach in distributed file system for write optimization.

In our case, we observed the write on line 3 is reordered to persist on server between line 6 and line 7 `[2]`.

Therefore, if RocksDB crashes right after line 5 `[1]`, only the first insert and third insert will be preserved on disk, and no sstfile is generated yet.

When restart from that disk state, RocksDB will recover the first and third insert, but lose data for the second insert, which causes the hole recovery problem described in this [post](http://rocksdb.org/blog/2022/10/05/lost-buffered-write-recovery.html).

### Expected behavior
RocksDB should recover with either of following state,
- `null`
- `insert1`
- `insert1 insert2`
- `insert1 insert2 insert3`

### Actual behavior
RocksDB recovers with `insert1 insert3`.

### Steps to reproduce the behavior
One way to reproduce the behavior without setting up GlusterFS is to run a simple workload that does at least 3 inserts to RocksDB, then construct a crash state with the write for second insert get reordered after third insert. Since there is indeterminism with the log compaction process due to concurrency nature, to replay the bug, we should make sure the write for third insert is finished before the data on first two inserts is converted to sstfile.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.