facebook / facebook/rocksdb

compaction gets stuck forever on L0->base during write burst

Open
#4,991 6 comments 0 reactions 0 assignees View on GitHub
up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

RocksDB gets stuck in L0->L0 and L0->base compactions in very write-heavy benchmarks. base->base+1 almost never happens and base level's score is usually reported as zero due to pending L0->base.

It is caused by interactions between:

(1) intra-L0 compaction,
(2) Siying's optimization to use L0 size as base level target size in write-heavy scenario, and
(3) An existing workaround to disable base->base+1 compaction when L0 is eligible for compaction but not scheduled.

(3) has existed the longest but does not seem particularly relevant in these modern times, where we can keep doing intra-L0 while base level is contended. I saw more benefit than expected by removing it, though cannot explain why yet.

Benchmark command:

```
$ TEST_TMPDIR=/data/compaction_bench ./db_bench -benchmarks=filluniquerandom -num=50000000 -max_write_buffer_number=4 -rate_limiter_bytes_per_sec=41943040 -write_buffer_size=2097152 -target_file_size_base=262072 -max_bytes_for_level_base=4194304 -compression_type=none -max_background_jobs=3 -level_compaction_dynamic_level_bytes=true -target_file_size_multiplier=2 -level0_file_num_compaction_trigger=2 -stats_per_interval=1 -stats_interval_seconds=10
```

Results with (3):

```
filluniquerandom : 24.812 micros/op 40303 ops/sec; 4.5 MB/s
```

```
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 23/8 347.12 MB 53.6 12.9 0.0 12.9 18.6 5.7 0.0 3.3 16.8 24.1 789.66 82.13 3775 0.209 112M 0
L4 5835/5835 1.44 GB 0.0 16.2 3.8 12.4 16.2 3.8 0.0 4.3 21.9 21.9 755.57 225.09 11 68.689 141M 0
L5 11715/0 2.91 GB 2.9 5.4 1.8 3.7 5.4 1.8 1.9 3.1 23.6 23.6 235.62 61.19 542 0.435 47M 0
L6 2336/0 986.33 MB 0.0 1.2 1.0 0.2 1.2 1.0 0.0 1.2 14.7 14.7 81.88 9.90 534 0.153 10M 0
Sum 19909/5843 5.65 GB 0.0 35.7 6.5 29.2 41.3 12.1 1.9 7.3 19.6 22.7 1862.73 378.31 4862 0.383 312M 0
Int 0/0 0.00 KB 0.0 0.3 0.0 0.3 0.4 0.1 0.0 4.1 15.5 20.4 18.96 1.89 62 0.306 2500K 0
```

Results without (3):

```
filluniquerandom : 22.113 micros/op 45222 ops/sec; 5.0 MB/s
```

```
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 22/13 930.28 MB 184.6 13.9 0.0 13.9 19.5 5.6 0.0 3.4 17.3 24.4 819.79 85.65 3825 0.214 121M 0
L4 15106/15106 3.75 GB 0.0 17.7 3.8 13.9 17.7 3.8 0.0 4.6 22.4 22.4 806.52 238.72 13 62.040 154M 0
L5 1865/0 472.58 MB 0.3 2.2 0.8 1.4 2.2 0.8 0.1 2.9 21.9 21.9 102.76 23.15 6 17.126 19M 0
L6 1533/0 535.67 MB 0.0 0.7 0.5 0.2 0.7 0.5 0.0 1.4 15.4 15.4 48.60 6.37 423 0.115 6395K 0
Sum 18526/15119 5.64 GB 0.0 34.5 5.1 29.4 40.1 10.7 0.1 7.1 19.9 23.1 1777.67 353.89 4267 0.417 301M 0
Int 0/0 0.00 KB 0.0 0.2 0.0 0.2 0.3 0.1 0.0 3.1 14.6 21.4 14.65 1.46 57 0.257 1821K 0
```

I'd also speculate that when (2) is active, we should be calculate L0 compaction score using file count only, i.e., do not take into account L0 size.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.