Potential crash-consistency vulnerability when allow_mmap_writes is enabled
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
Hello RocksDB developers,
We've found a potential crash-consistency vulnerability for RocksDB when `allow_mmap_writes` is forced to be `true`. We simulated a crash during the process of `Append` calls on `PosixMmapFile` using a crash-consistency bug detection tool that we are currently building. And then when we re-open the database it returns an assertion failure: "db/db_impl/db_impl.cc:1122: void rocksdb::DBImpl::DumpStats(): Assertion `property_info != nullptr' failed."
An important change worth mentioning is that we observed `use_mmap_writes` will be turned off in function [OptimizeForLogWrite](https://github.com/facebook/rocksdb/blob/main/env/fs_posix.cc#L910) and [OptimizeForManifestWrite](https://github.com/facebook/rocksdb/blob/main/env/fs_posix.cc#L925). So we changed these two functions to instead set `use_mmap_writes` to be `true`. We do not know if RocksDB's persistent mechanisms are built upon the assumption that these two optimization functions will always be invoked and thus `use_mmap_writes` is always false.
### Expected behavior
The database should re-open without errors.
### Actual behavior
The database reports an assertion failure "db/db_impl/db_impl.cc:1122: void rocksdb::DBImpl::DumpStats(): Assertion `property_info != nullptr' failed." and aborted.
### Steps to reproduce the behavior
1. Compile the following test case which create a database and insert a key
```
# workload.cpp
#include
#include
#include "rocksdb/db.h"
#include
#include
#include
#include
#include "common.h"
using namespace std;
using namespace rocksdb;
int main(int argc, char *argv[]) {
if (argc != 2) {
printf("Usage: %s \n", argv[0]);
exit(1);
}
/* Variable declarations and some setup */
DB* db;
Options options;
Status ret;
WriteOptions write_options;
string key, value;
int i;
options.create_if_missing = true;
options.allow_mmap_writes = true;
// Set other RocksDB-specific options as needed
write_options.sync = true;
/* Open the database */
ret = DB::Open(options, argv[1], &db);
// assert(ret.ok());
if (!ret.ok()) {
printf("Open failed\n");
printf("%s\n", ret.ToString().c_str());
exit(1);
}
/* Put one row into the database */
key = string(gen_string(0, KEY_SIZE));
value = string(gen_string(0, VALUE_SIZE));
ret = db->Put(write_options, key, value);
assert(ret.ok());
/* Close the database */
delete db;
}
```
2. Create an empty directory `testdb`
3. Use gdb to run the following test case by `gdb workload testdb`
4. Put a breakpoint in `Append` function of `PosixMmapFile` in gdb by `br env/io_posix.cc:1147`
5. Run the program until the breakpoint and quit it to simulate a crash
6. Compile and run the following minimized crash-recovery program which open a database and create an iterator, the assertion failure should be observed
```
# checker.cpp
#include
#include
#include "rocksdb/db.h"
#include
#include
#include
#include
#include "common.h"
#include
#include
#include
using namespace std;
using namespace rocksdb;
int main(int argc, char *argv[]) {
if (argc != 2) {
printf("Usage: %s \n", argv[0]);
exit(1);
}
/* Variable declarations and some setup */
DB* db;
Options options;
Status ret;
ReadOptions read_options;
string key, value;
int i;
Iterator* it;
char printed_messages[1000];
int fd, pos;
int retreived_rows = 0;
int row_present[10] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
char db_path[10000];
options.create_if_missing = true;
read_options.verify_checksums = true;
strcpy(db_path, argv[1]);
// strcat(db_path, "/testdb");
ret = DB::Open(options, db_path, &db);
// assert(ret.ok());
if (!ret.ok()) {
printf("Open failed\n");
printf("%s\n", ret.ToString().c_str());
exit(1);
}
it = db->NewIterator(read_options);
// assert(it->status().ok(), "Iterator creation failed");
if (!it->status().ok()) {
printf("Iterator creation failed\n");
printf("%s\n", it->status().ToString().c_str());
exit(1);
}
}
```
### RocksDB version
tag: v8.10.0
### Linux distribution
Ubuntu 22.04.2 LTS
### Filesystem version
ext4
Contributor guide
Research direction
Start with OptimizeForLogWrite and OptimizeForManifestWrite in env/fs_posix.cc and PosixMmapFile::Append in env/io_posix.cc. Reproduce the crash using the supplied workload.cpp and checker.cpp, then inspect the assertion at db/db_impl/db_impl.cc:1122 during recovery. Done means the database reopens without errors or assertion failure when mmap writes are enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- databases, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100