facebook / facebook/rocksdb

db_bench --benchmarks=waitforcompaction hangs with universal compaction

Open
#9,275 1 comment 0 reactions 1 assignee Claimed by @mdcallag View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

### Expected behavior

With --benchmarks=waitforcompaction the test should return once compaction is not in progress or pending.

### Actual behavior

There are intermittent hangs.

### Steps to reproduce the behavior

db_bench --benchmarks=fillseq,overwrite,waitforcompaction

The problem occurs when db_bench [uses kCompactionPending](https://github.com/facebook/rocksdb/blob/dc5de45af8fc1bcc90edea5cd9a5695bd85a8baf/tools/db_bench_tool.cc#L7773) which triggers a [call to HandleCompactionPending](https://github.com/facebook/rocksdb/blob/01bd86ad356a69ab54783aaaa198b6faa5684fb1/db/internal_stats.cc#L453) which [calls NeedsCompaction](https://github.com/facebook/rocksdb/blob/01bd86ad356a69ab54783aaaa198b6faa5684fb1/db/internal_stats.cc#L1033) which [returns true](https://github.com/facebook/rocksdb/blob/c712b68f5bdf2c3ee9b71c6bec599f1451cd806f/db/compaction/compaction_picker_universal.cc#L278) when compaction_score is >= 1 for universal.

This eventually leads to a call to PickCompaction because the read-amp trigger means compaction is needed. But if the sizes of the sorted runs are such that a compaction is not chosen (the next sorted run is too much larger than the current sorted run) then it ends [up here](https://github.com/facebook/rocksdb/blob/c712b68f5bdf2c3ee9b71c6bec599f1451cd806f/db/compaction/compaction_picker_universal.cc#L440) and the problem is the code that does:
```
num_sr_not_compacted >
mutable_cf_options_.level0_file_num_compaction_trigger
```

This only triggers a call to PickCompactionToReduceSortedRuns when the number of sorted runs is greater than L0_file_num_compaction_trigger. However it should be triggered when that number is greater than or equal.

With the current code, we can end up with compaction_score = 1 (because #sorted runs == L0 compaction trigger) but compaction won't get done. So waitforcompaction loops waiting for compaction_score to be less than 1 but that never happens.

Note that if the > is changed to >= then the value of num_files should be incremented by 1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.