cockroachdb / cockroachdb/pebble
db: decide on the correct shape for large LSMs
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
We currently use the code below to calculate a smoothed level multiplier:
```go
bottomLevelSize := dbSize - dbSize/uint64(p.opts.Experimental.LevelMultiplier)
...
smoothedLevelMultiplier := 1.0
if p.baseLevel < numLevels-1 {
smoothedLevelMultiplier = math.Pow(
float64(bottomLevelSize)/float64(baseBytesMax),
1.0/float64(numLevels-p.baseLevel-1))
}
levelSize := float64(baseBytesMax)
for level := p.baseLevel; level < numLevels; level++ {
if level > p.baseLevel && levelSize > 0 {
levelSize *= smoothedLevelMultiplier
}
...
}
```
At the CRDB settings (baseBytesMax=64MiB, LevelMultiplier=10), once the LSM size goes over 7TiB, the smothedLevelMultiplier goes [above 10](https://www.google.com/search?q=%28%287*0.9%29+TiB+%2F+64+MiB%29%5E%281%2F5%29). At the current advisory limit of 10TiB, it is [10.8](https://www.google.com/search?q=%28%2810*0.9%29+TiB+%2F+64+MiB%29%5E%281%2F5%29). At a hypothetical 40TiB, it is [14.3](https://www.google.com/search?q=%28%2840*0.9%29+TiB+%2F+64+MiB%29%5E%281%2F5%29).
Instead of letting the multiplier go above the `LevelMultiplier`, we should increase the target for L1.
It could be something like this:
```
if smoothedLevelMultiplier > float64(p.opts.Experimental.LevelMultiplier) {
levelSize = float64(curLevelSize)
smoothedLevelMultiplier = float64(p.opts.Experimental.LevelMultiplier)
}
```
Jira issue: PEBBLE-1296
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.