facebook / facebook/rocksdb

Intra0 compaction cause big write stall

Open
#10,903 3 comments 0 reactions 1 assignee Claimed by @ajkr View on GitHub
design discussion
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

I have already raised a related question in the RocksDB dev group:https://groups.google.com/g/rocksdb/c/9X7L765lKik
**Problem Description:**

Intral0compaction may cause a big write stall.
```
There is a SetupInitialFiles() function in LeveledCompactionBuilder with the following logic:
if (PickFileToCompact()) {
......
}else{
......
if(PickIntraL0Compaction())
......
}
```
Under write-heavy workload, the existence of intraL0compaction will make level 0 always have some files' being_compacted to be true, which will make PickFileToCompact() false, and then continue to enter the logic of PickIntraL0Compaction(), which is an infinite loop. Finally, when the level 0 file reaches a large value, the compaction of level 0 to level 1 will be performed, which will cause a large write pause, which is almost as long as 50s in my test.

**Reply To Problem :**
Thanks for mdcallag@gmail.com told me that the setting of the max_compaction_bytes parameter has been adopted in https://github.com/facebook/rocksdb/pull/5299, and try to prevent the intra0 compaction when there are too big bytes involved, so that the level 0 to level 1 compaction is no longer block by intral0compaction.

**Further Questions:**
I went to read the code of max_compaction_bytes application at other levels except level 0. Its function seems is - "In a sub compaction, if the size of the currently constructed sst plus the overlap size of the grandparent's sst exceeds max_compaction_bytes, the construction of this sst will be stopped in advance". This essentially avoids a large compaction in the future.

But I think this is fundamentally different from its role in level 0. Because the compaction of level n to level n+1, even if it is large, it will not block other compactions of level n to level n+1, as long as there is no overlapping key between them. But considering that level 0 is out of order, if there is a compaction in progress at level 0, it will block all compactions of level 0 to level 1.

And in level 0, this parameter is almost closely related to the progress of intra0compaction. The size of level 0 often needs to reach the size of max_compaction_bytes to avoid intra0 compaction, and then level 0 to level 1 compaction can be performed. Its role at level 0 is closer to the upper limit of the file size of a level 0, which is essentially different from its role in compaction at other levels.

Sometimes, we may want to maintain a small file size at level 0, but we are not very concerned about the file size that may be involved in compaction at other levels.
**Maybe we can split this option into two options? For level 0 and level others, respectively.**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.