[BUG] thread hangs forever when two Flush races
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
### Steps to reproduce the behavior
necessary conditions for this bug
1. two(multiple) cfs
2. min_write_buffer_number_to_merge > 1
3. two concurrent Flush calls(on the same cf)
here is the cpptest to reproduce, I've tested it still exists on v8.8.1
t1 will join, but t2 hangs there forever.
```
TEST_F(DBFlushTest, ManualFlushWithMinWriteBufferNumberToMergeHang) {
Options options = CurrentOptions();
options.write_buffer_size = 100;
options.max_write_buffer_number = 4;
options.min_write_buffer_number_to_merge = 3;
options.max_background_jobs = 4; // so max_flush_jobs = 1
Reopen(options);
CreateColumnFamilies({"other_cf"}, options);
std::atomic sync_logs_cnt{0}, flush_scheduled{0};
SyncPoint::GetInstance()->SetCallBack(
"DBImpl::SyncClosedLogs:Start", [&](void *arg) {
(void)arg;
sync_logs_cnt.fetch_add(1);
std::cout<< "sync_logs_cnt inc " << sync_logs_cnt.load() << std::endl;
while (flush_scheduled.load() == 1) {
sleep(1);
}
});
SyncPoint::GetInstance()->SetCallBack(
"DBImpl::FlushMemTable:AfterScheduleFlush", [&](void *arg) {
(void)arg;
flush_scheduled.fetch_add(1);
std::cout << "flush_scheduled inc " << flush_scheduled.load() << std::endl;
});
SyncPoint::GetInstance()->EnableProcessing();
port::Thread t1([&]() {
ASSERT_OK(Put("key1", "value1"));
ASSERT_OK(Flush());
});
port::Thread t2([&]() {
while (sync_logs_cnt.load() != 1) {
sleep(1);
}
ASSERT_OK(Put("key2", "value2"));
ASSERT_OK(Flush());
});
t1.join();
std::cout << "t1 joined" << std::endl;
t2.join();
std::cout << "t2 joined" << std::endl;
}
```
### In One Sentence
when two Flush(with wait=true) races, the boolean value flush_requested_ is set to false by the first thread, causing the FlushRequest from the second thread being ingored, because IsFlushPending returns false. While the second thread is hanging on WaitForFlushMemTables because the immtable switched by it is really not flushed.
### Explanations
1. t2 blocks until t1 flushes and enter here https://github.com/facebook/rocksdb/blob/986b8b9f20893dec811d8ecdb97b6a47f20d322d/db/db_impl/db_impl_compaction_flush.cc#L269
at this point, t1 has constructed the FlushJob but has not `PickMemtables` to flush. t1 released dbmutex to SyncClosedLogs.
2. then t2 moves on, t1 hangs until t2 enqueues flush_queue_
t2 has a chance to enqueue flush_queue_ because dbmutex is released by t1,
https://github.com/facebook/rocksdb/blob/986b8b9f20893dec811d8ecdb97b6a47f20d322d/db/db_impl/db_impl_compaction_flush.cc#L2967
3. when t1 sees t2 has published its flush_req into flush_queue_, t1 moves on to PickMemtables. although there are two immutables memtables, one is contributed by t1.Flush, one is contributed by t2.FLush. t1 will pick only one memtable to flush because when t1 sets max_memtable_id for its FlushJOb, t2 has not yet run. see https://github.com/facebook/rocksdb/blob/986b8b9f20893dec811d8ecdb97b6a47f20d322d/db/db_impl/db_impl_compaction_flush.cc#L226
4. a boolean value imm.flush_requested_ is set to false after t1 picks one imm to flush, see https://github.com/facebook/rocksdb/blob/986b8b9f20893dec811d8ecdb97b6a47f20d322d/db/memtable_list.cc#L431-L433
5. when t1 finishes the FlushJob and try to find more, it will find no more tasks, IsFlushPending returns false https://github.com/facebook/rocksdb/blob/986b8b9f20893dec811d8ecdb97b6a47f20d322d/db/db_impl/db_impl_compaction_flush.cc#L3192 flush_requested_ = false, min_write_buffer_number_to_merge = 3, num_flush_not_started_ = 1 https://github.com/facebook/rocksdb/blob/986b8b9f20893dec811d8ecdb97b6a47f20d322d/db/memtable_list.cc#L370-L377
### SeqGraph
```
t1 t2
Flush() called
... now one imm(id = 1)
FlushMemTableToOutputFile
set max_memtable_id=1
dbmutex.Unlock
Flush called
... now two imms(id = 1,2)
dbmutex.Lock
flush_request_.enqueue()
dbmutex.Unlock
dbmutex.Lock
PickMemtables(only 1 picked)
imm.flush_request_ true->false
try pick but find nomore to flush
hangs on WaitForFlushMemTables
```
to make the seq simpler, I mixed FlushThread and thread t1.
### How to fix
TODO
Contributor guide
Assessment
This issue has not been assessed yet.