Disk space used by the DB grows without bounds
- Dominant language
- C++
- Stars
- 39.4k
- Forks
- 8.2k
- PR merge metrics
- No merged PRs in 30d
Description
In Apache Mesos we use LevelDB as an underlying storage for our replicated log storage implementation. We use the default `BytewiseComparator` and use log positions converted to fixed-length strings (`sprintf(key, "%.10d", position)`) as LevelDB keys. Stored value size is typically ~9KB, max ~500KB. Our usage pattern looks something like this:
1. Sequentially write entries with keys 1...N (log appends, followed by a snapshot and a truncation action, N > 1000)
2. Batch delete entries 1...N-1 (log truncation)
3. Sequentially write entries with keys N+1...N\*2
4. Batch delete entries N+1...N\*2-1
5. Repeat indefinitely
With such usage the amount of disk space used by LevelDB keeps growing over time until it eats all space on the partition despite the fact that at any time the DB should be storing more or less the same amount of data. It seems that we never meet the conditions for the background compaction to kick in and throw deleted keys away and the only way to reclaim the disk space is to call `DB::CompactRange()` after keys deletion. Is that correct and the described behavior is expected or it should not work like that?
This happens even when N = 3 (put 1, put 2, put 3, delete 1, put 4, put 5, delete 2-3, ...), but in that case stored value size is ~15MB.
Contributor guide
Assessment
This issue has not been assessed yet.