OnFlushComplete may be called out of order
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
### Expected behavior
`OnFlushComplete` should be called in the order of sealed.
### Actual behavior
It may be called in chaos.
### Steps to reproduce the behavior
Keep loading data in rocksdb. I install a listener to listen for both `OnMemtableSealed` event and `OnFlushComplete` event. I record the earliest seqno from sealed and largest seqno from flush. Every time a largest seqno is received, any earliest seqno smaller than it will be removed. If `OnFlushComplete` comes in the same order as `OnMemtableSealed`, then there always be at least one earliest seqno found. However, I did see the constraint failed in rare cases. The log shows that there is a Flush finishes out of order in that cases:
```
[2023/02/02 20:37:53.762 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 385] Level-0 flush table #61020: 32858346 bytes OK
[2023/02/02 20:37:54.047 +08:00][5][INFO] [251_13119688](Original Log Time 2023/02/02-20:37:54.009792) [db/db_impl/db_impl_compaction_flush.cc:301] [lock] Level summary: base level 6 level multiplier 10.00 max bytes base 134217728 files[1 0 0 0 0 0 4] max score 1.00
[2023/02/02 20:37:54.751 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 388] Level-0 flush table #61034: 21214984 bytes OK
[2023/02/02 20:37:54.751 +08:00][5][INFO] [251_13119688](Original Log Time 2023/02/02-20:37:54.751310) [db/db_impl/db_impl_compaction_flush.cc:301] [lock] Level summary: base level 6 level multiplier 10.00 max bytes base 134217728 files[0 0 0 0 0 0 5] max score 0.00
[2023/02/02 20:37:54.899 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 387] Level-0 flush table #61029: 45351480 bytes OK
[2023/02/02 20:37:55.059 +08:00][5][INFO] [251_13119688][db/flush_job.cc:946] [lock] [JOB 389] Level-0 flush table #61037: 19563984 bytes OK
```
Any thought? Should I assume `OnFlushComplete` is always called in the sealed order? How to verify it's expected chaos or there is potential bugs (like later sealed memtable get install earlier)?
Contributor guide
Assessment
This issue has not been assessed yet.