facebook / facebook/rocksdb

OnFlushComplete may be called out of order

Open
#11,183 7 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

### Expected behavior

`OnFlushComplete` should be called in the order of sealed.

### Actual behavior

It may be called in chaos.

### Steps to reproduce the behavior

Keep loading data in rocksdb. I install a listener to listen for both `OnMemtableSealed` event and `OnFlushComplete` event. I record the earliest seqno from sealed and largest seqno from flush. Every time a largest seqno is received, any earliest seqno smaller than it will be removed. If `OnFlushComplete` comes in the same order as `OnMemtableSealed`, then there always be at least one earliest seqno found. However, I did see the constraint failed in rare cases. The log shows that there is a Flush finishes out of order in that cases:
```
[2023/02/02 20:37:53.762 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 385] Level-0 flush table #61020: 32858346 bytes OK
[2023/02/02 20:37:54.047 +08:00][5][INFO] [251_13119688](Original Log Time 2023/02/02-20:37:54.009792) [db/db_impl/db_impl_compaction_flush.cc:301] [lock] Level summary: base level 6 level multiplier 10.00 max bytes base 134217728 files[1 0 0 0 0 0 4] max score 1.00
[2023/02/02 20:37:54.751 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 388] Level-0 flush table #61034: 21214984 bytes OK
[2023/02/02 20:37:54.751 +08:00][5][INFO] [251_13119688](Original Log Time 2023/02/02-20:37:54.751310) [db/db_impl/db_impl_compaction_flush.cc:301] [lock] Level summary: base level 6 level multiplier 10.00 max bytes base 134217728 files[0 0 0 0 0 0 5] max score 0.00
[2023/02/02 20:37:54.899 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 387] Level-0 flush table #61029: 45351480 bytes OK
[2023/02/02 20:37:55.059 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 389] Level-0 flush table #61037: 19563984 bytes OK
```
Any thought? Should I assume `OnFlushComplete` is always called in the sealed order? How to verify it's expected chaos or there is potential bugs (like later sealed memtable get install earlier)?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.