facebook / facebook/rocksdb

Potential crash-consistency vulnerability when allow_mmap_writes is enabled

Open
#12,250 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

Hello RocksDB developers,

We've found a potential crash-consistency vulnerability for RocksDB when `allow_mmap_writes` is forced to be `true`. We simulated a crash during the process of `Append` calls on `PosixMmapFile` using a crash-consistency bug detection tool that we are currently building. And then when we re-open the database it returns an assertion failure: "db/db_impl/db_impl.cc:1122: void rocksdb::DBImpl::DumpStats(): Assertion `property_info != nullptr' failed."

An important change worth mentioning is that we observed `use_mmap_writes` will be turned off in function [OptimizeForLogWrite](https://github.com/facebook/rocksdb/blob/main/env/fs_posix.cc#L910) and [OptimizeForManifestWrite](https://github.com/facebook/rocksdb/blob/main/env/fs_posix.cc#L925). So we changed these two functions to instead set `use_mmap_writes` to be `true`. We do not know if RocksDB's persistent mechanisms are built upon the assumption that these two optimization functions will always be invoked and thus `use_mmap_writes` is always false.

### Expected behavior

The database should re-open without errors.

### Actual behavior

The database reports an assertion failure "db/db_impl/db_impl.cc:1122: void rocksdb::DBImpl::DumpStats(): Assertion `property_info != nullptr' failed." and aborted.

### Steps to reproduce the behavior

1. Compile the following test case which create a database and insert a key

```

# workload.cpp

#include
#include
#include "rocksdb/db.h"
#include
#include
#include
#include
#include "common.h"

using namespace std;
using namespace rocksdb;

int main(int argc, char *argv[]) {
if (argc != 2) {
printf("Usage: %s \n", argv[0]);
exit(1);
}
/* Variable declarations and some setup */
DB* db;
Options options;
Status ret;
WriteOptions write_options;
string key, value;
int i;

options.create_if_missing = true;
options.allow_mmap_writes = true;
// Set other RocksDB-specific options as needed
write_options.sync = true;

/* Open the database */
ret = DB::Open(options, argv[1], &db);
// assert(ret.ok());
if (!ret.ok()) {
printf("Open failed\n");
printf("%s\n", ret.ToString().c_str());
exit(1);
}

/* Put one row into the database */
key = string(gen_string(0, KEY_SIZE));
value = string(gen_string(0, VALUE_SIZE));
ret = db->Put(write_options, key, value);
assert(ret.ok());
/* Close the database */
delete db;
}

```

2. Create an empty directory `testdb`
3. Use gdb to run the following test case by `gdb workload testdb`
4. Put a breakpoint in `Append` function of `PosixMmapFile` in gdb by `br env/io_posix.cc:1147`
5. Run the program until the breakpoint and quit it to simulate a crash
6. Compile and run the following minimized crash-recovery program which open a database and create an iterator, the assertion failure should be observed

```

# checker.cpp

#include
#include
#include "rocksdb/db.h"
#include
#include
#include
#include
#include "common.h"
#include
#include
#include

using namespace std;
using namespace rocksdb;

int main(int argc, char *argv[]) {
if (argc != 2) {
printf("Usage: %s \n", argv[0]);
exit(1);
}
/* Variable declarations and some setup */
DB* db;
Options options;
Status ret;
ReadOptions read_options;
string key, value;
int i;

Iterator* it;
char printed_messages[1000];
int fd, pos;
int retreived_rows = 0;
int row_present[10] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
char db_path[10000];

options.create_if_missing = true;
read_options.verify_checksums = true;

strcpy(db_path, argv[1]);
// strcat(db_path, "/testdb");

ret = DB::Open(options, db_path, &db);
// assert(ret.ok());
if (!ret.ok()) {
printf("Open failed\n");
printf("%s\n", ret.ToString().c_str());
exit(1);
}

it = db->NewIterator(read_options);
// assert(it->status().ok(), "Iterator creation failed");
if (!it->status().ok()) {
printf("Iterator creation failed\n");
printf("%s\n", it->status().ToString().c_str());
exit(1);
}

}

```

### RocksDB version

tag: v8.10.0

### Linux distribution

Ubuntu 22.04.2 LTS

### Filesystem version

ext4

Contributor guide

Open the contributing guide

Research direction

Start with OptimizeForLogWrite and OptimizeForManifestWrite in env/fs_posix.cc and PosixMmapFile::Append in env/io_posix.cc. Reproduce the crash using the supplied workload.cpp and checker.cpp, then inspect the assertion at db/db_impl/db_impl.cc:1122 during recovery. Done means the database reopens without errors or assertion failure when mmap writes are enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
databases, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.