facebook / facebook/rocksdb

Write-amp increased for fillseq with universal compaction

Open
#10,082 4 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

This occurs after 4.1 but in or before 5.1.4.
* Throughput is ~1.1X better for 4.1
* Write-amp is ~1.4X larger for 5.1.4
* Compaction wall clock seconds are ~1.7X larger for 5.1.4

Notice in the compaction IO stats below that with v4.1 there is data in L0 and L7. While with v5.1.4 there is data in L0, L2, L3, L4, L5, L6. For v5.1.4 only trivial move is used for L2 through L6. But with v5.1.4 the amount of compaction within L0 is larger -- see the _Write(GB)_ column in the compaction IO stats.

I don't know whether the issue is from the extra compaction in L0 or from the extra trivial moves from the additional levels. Also see issues [10075](https://github.com/facebook/rocksdb/issues/10075) and [9423](https://github.com/facebook/rocksdb/issues/9423).

Perf for 4.1
```
fillseq : 0.844 micros/op 1184197 ops/sec; 474.3 MB/s
Microseconds per op:
Count: 4000000000 Average: 0.8445 StdDev: 13.15
Min: 0.0000 Median: 0.5152 Max: 14651.0000
Percentiles: P50: 0.52 P75: 0.77 P99: 2.37 P99.9: 5.40 P99.99: 484.37
```

Perf for 5.1.4
```
fillseq : 0.937 micros/op 1067365 ops/sec; 427.5 MB/s
Microseconds per write:
Count: 4000000000 Average: 0.9369 StdDev: 0.91
Min: 0 Median: 0.5115 Max: 3390828
Percentiles: P50: 0.51 P75: 0.77 P99: 2.41 P99.9: 5.75 P99.99: 795.25
```

Compaction IO stats at test end for 4.1
```
Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 13/12 109.92 0.0 870.1 0.0 870.1 1741.4 871.3 0.0 0.0 65.4 130.9 13618 123252 0.110 16485 3973M 0
L7 17861/0 892122.30 0.0 0.0 0.0 0.0 0.0 0.0 871.2 0.0 0.0 0.0 0 0 0.000 0 0 0
Sum 17874/12 892232.22 0.0 870.1 0.0 870.1 1741.4 871.3 871.2 2.0 65.4 130.9 13618 123252 0.110 16485 3973M 0
Int 0/0 0.00 0.0 5.0 0.0 5.0 10.1 5.0 5.1 2.0 64.9 129.8 80 714 0.112 37 23M 0
Flush(GB): cumulative 871.406, interval 5.045
Stalls(count): 16485 level0_slowdown, 16485 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 pending_compaction_bytes, 0 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard

** DB Stats **
Uptime(secs): 3360.3 total, 20.0 interval
Cumulative writes: 3979M writes, 3979M keys, 3979M batches, 1.0 writes per batch, ingest: 1616.01 GB, 492.45 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative compaction: 1741.42 GB write, 530.67 MB/s write, 870.10 GB read, 265.15 MB/s read, 13618.3 seconds
Cumulative stall: 00:05:6.599 H:M:S, 9.1 percent
```

Compaction IO stats at test end for 5.1.4
```
evel Files Size(MB} Score Read(GB} Rn(GB} Rnp1(GB} Write(GB} Wnew(GB} Moved(GB} W-Amp Rd(MB/s} Wr(MB/s} Comp(sec} Comp(cnt} Avg(sec} KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 8/6 118.78 0.0 1614.1 0.0 1614.1 2491.5 877.4 0.0 0.0 73.3 113.1 22553 178836 0.126 7346M 0
L2 115/0 3978.77 0.0 0.0 0.0 0.0 0.0 0.0 758.6 0.0 0.0 0.0 0 0 0.000 0 0
L3 2172/0 64357.19 0.0 0.0 0.0 0.0 0.0 0.0 633.5 0.0 0.0 0.0 0 0 0.000 0 0
L4 2563/0 70176.73 0.0 0.0 0.0 0.0 0.0 0.0 419.9 0.0 0.0 0.0 0 0 0.000 0 0
L5 2506/0 77268.72 0.0 0.0 0.0 0.0 0.0 0.0 366.2 0.0 0.0 0.0 0 0 0.000 0 0
L6 6849/0 180472.26 0.0 0.0 0.0 0.0 0.0 0.0 395.3 0.0 0.0 0.0 0 0 0.000 0 0
Sum 30569/6 898441.39 0.0 1614.1 0.0 1614.1 2491.5 877.4 3063.6 2.8 73.3 113.1 22553 178836 0.126 7346M 0
Uptime(secs): 3740.2 total, 3740.2 interval
Flush(GB): cumulative 877.468, interval 3.960
Cumulative compaction: 2491.51 GB write, 682.13 MB/s write, 1614.13 GB read, 441.92 MB/s read, 22553.0 seconds
Stalls(count): 2088 level0_slowdown, 2088 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 3 memtable_compaction, 129 memt
able_slowdown, interval 0 total count

** DB Stats **
Uptime(secs): 3740.2 total, 20.0 interval
Cumulative writes: 3993M writes, 3993M keys, 3993M commit groups, 1.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:55.260 H:M:S, 1.5 percent
```

Command lines for 4.1 and 5.1.4
```
# for 4.1
./db_bench --benchmarks=fillseq --level0_file_num_compaction_trigger=8 --level0_slowdown_writes_trigger=20 --level0_stop_writes_trigger=30 --max_background_flushes=4 --max_background_compactions=12 --max_write_buffer_number=8 --db=/data/m/rx --wal_dir=/data/m/rx --num=4000000000 --num_levels=8 --key_size=20 --value_size=400 --block_size=8192 --cache_size=225485783040 --cache_numshardbits=6 --compression_ratio=0.5 --compression_type=lz4 --bytes_per_sync=8388608 --cache_index_and_filter_blocks=1 --benchmark_write_rate_limit=0 --writes_per_second=0 --write_buffer_size=16777216 --target_file_size_base=16777216 --max_bytes_for_level_base=67108864 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=20 --report_interval_seconds=5 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --subcompactions=4 --compaction_style=1 --universal_compression_size_percent=80 --universal_min_merge_width=2 --universal_max_merge_width=20 --universal_size_ratio=1 --universal_max_size_amplification_percent=200 --universal_allow_trivial_move=1 --universal_compression_size_percent=-1 --use_existing_db=0 --sync=0 --threads=1 --memtablerep=vector --disable_wal=1 --seed=1649938008 --report_file=bm.uc.nt32.cm1.d0.sc4.tm/v4.1/benchmark_fillseq.wal_disabled.v400.log.r.csv

# for 5.1.4
./db_bench --benchmarks=fillseq --allow_concurrent_memtable_write=false --level0_file_num_compaction_trigger=8 --level0_slowdown_writes_trigger=20 --level0_stop_writes_trigger=30 --max_background_flushes=4 --max_background_compactions=12 --max_write_buffer_number=8 --db=/data/m/rx --wal_dir=/data/m/rx --num=4000000000 --num_levels=8 --key_size=20 --value_size=400 --block_size=8192 --cache_size=225485783040 --cache_numshardbits=6 --compression_max_dict_bytes=0 --compression_ratio=0.5 --compression_type=lz4 --bytes_per_sync=8388608 --cache_index_and_filter_blocks=1 --cache_high_pri_pool_ratio=0.5 --benchmark_write_rate_limit=0 --write_buffer_size=16777216 --target_file_size_base=16777216 --max_bytes_for_level_base=67108864 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=20 --report_interval_seconds=5 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --subcompactions=4 --compaction_style=1 --universal_compression_size_percent=80 --pin_l0_filter_and_index_blocks_in_cache=1 --universal_min_merge_width=2 --universal_max_merge_width=20 --universal_size_ratio=1 --universal_max_size_amplification_percent=200 --universal_allow_trivial_move=1 --universal_compression_size_percent=-1 --use_existing_db=0 --sync=0 --threads=1 --memtablerep=vector --allow_concurrent_memtable_write=false --disable_wal=1 --seed=1649982947 --report_file=bm.uc.nt32.cm1.d0.sc4.tm/v5.1.4/benchmark_fillseq.wal_disabled.v400.log.r.csv
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied db_bench fillseq commands for RocksDB 4.1 and 5.1.4, then compare the universal-compaction IO statistics. Investigate whether the regression comes from extra L0 compaction or trivial moves in L2-L6, using issues 10075 and 9423 for context. Done means identifying the cause and validating a fix against the reported write amplification and compaction costs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.