facebook / facebook/rocksdb

Segfault on iterator seek

Open
#10,866 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

We are using v7.7.3 and encounter segfault on rocksdb::WBWIIteratorImpl::Seek intermittently.
The segfault only happens under heavy load, and even with heavy load it is not possible to trigger it reliably.
It is easier to trigger on Windows and MacOS Ventura, much harder on Linux and older MacOS versions.
It seems having more threads than CPU count is required to trigger it, but we are not sure.

The following is the trace on MacOS when RocksDB is compiled with debug symbols:

```
running 65 tests
Assertion failed: (x != nullptr), function FindGreaterOrEqual, file skiplist.h, line 312.
Assertion failed: (x != nullptr)Assertion failed: (x != nullptr), function FindGreaterOrEqual, f, function FindGreaterOrEqual, file skiplist.h, line 312.
Assertion failed: (x != nullptr), function FindGreaterOrEqual, file skiplist.h, line 312.
Assertion failed: (x != nullptr), function FindGreaterOrEqual, fAssertion failed: (x != nullptr)ile skiplist.h, line 312.
, function FindGreaterOrEqual, fAssertion failed: (x != nullptr)ile skiplist.h, line 312.
, function FindGreaterOrEqual, file skiplist.h, line 312.
ile skiplist.h, line 312.
warning: air_routes-68a595ab527559b8 was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 66602 stopped
* thread #16, name = 'dfw_by_region', stop reason = hit program assert
frame #4: 0x000000010080ab58 air_routes-68a595ab527559b8`rocksdb::SkipList::FindGreaterOrEqual(rocksdb::WriteBatchIndexEntry* const&) const (.cold.1) at skiplist.h:312:5 [opt]
309 int level = GetMaxHeight() - 1;
310 Node* last_bigger = nullptr;
311 while (true) {
-> 312 assert(x != nullptr);
313 Node* next = x->Next(level);
314 // Make sure the lists are sorted
315 assert(x == head_ || next == nullptr || KeyIsAfterNode(next->key, x));
Target 0: (air_routes-68a595ab527559b8) stopped.
(lldb) bt
* thread #16, name = 'dfw_by_region', stop reason = hit program assert
frame #0: 0x000000018e563224 libsystem_kernel.dylib`__pthread_kill + 8
frame #1: 0x000000018e599cec libsystem_pthread.dylib`pthread_kill + 288
frame #2: 0x000000018e4d32c8 libsystem_c.dylib`abort + 180
frame #3: 0x000000018e4d2620 libsystem_c.dylib`__assert_rtn + 272
* frame #4: 0x000000010080ab58 air_routes-68a595ab527559b8`rocksdb::SkipList::FindGreaterOrEqual(rocksdb::WriteBatchIndexEntry* const&) const (.cold.1) at skiplist.h:312:5 [opt]
frame #5: 0x000000010069854c air_routes-68a595ab527559b8`rocksdb::SkipList::FindGreaterOrEqual(this=, key=) const at skiplist.h:312:5 [opt]
frame #6: 0x000000010069b00c air_routes-68a595ab527559b8`rocksdb::WBWIIteratorImpl::Seek(rocksdb::Slice const&) [inlined] rocksdb::SkipList::Iterator::Seek(this=0x0000600000c0db40, target=0x0000000171cae280) at skiplist.h:252:18 [opt]
frame #7: 0x000000010069b000 air_routes-68a595ab527559b8`rocksdb::WBWIIteratorImpl::Seek(this=0x0000600000c0db30, key=) at write_batch_with_index_internal.h:240:21 [opt]
frame #8: 0x0000000100699398 air_routes-68a595ab527559b8`rocksdb::BaseDeltaIterator::Seek(this=0x0000600003710000, k=0x0000000171cae2f0) at write_batch_with_index_internal.cc:61:20 [opt]
```

We've also observed segfaults several lines down, for example on line 315 in skiplist.h,

```
assert(x == head_ || next == nullptr || KeyIsAfterNode(next->key, x));
```
`next->key` could point to rubbish.

Currently we are kind of stuck and not sure what steps we can take to produce more information about the crash.

Contributor guide

Open the contributing guide

Research direction

Start at WBWIIteratorImpl::Seek in write_batch_with_index_internal.h and write_batch_with_index_internal.cc, then inspect SkipList::Iterator::Seek and FindGreaterOrEqual in skiplist.h. Reproduce the failure under heavy load with more threads than CPUs, especially on Windows or macOS, using the reported assertions and stack trace. Done means identifying the cause of the invalid node access and adding a reliable regression test or fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.