facebook / facebook/rocksdb

wait_mutex duration reach 400ms when the number of total SST file reach 40 thousands

Open
#6,499 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

### Expected behavior
Write duration and get duration should not be impacted so much by compaction.

### Actual behavior
The compaction impacts the Get duration and the Write duration a lot when the number of total SST files is about 40K. After we open the perf_context feature, we found the wait_mutex duration is very large, this is the root cause for the high latency.

![perf-context](https://user-images.githubusercontent.com/19700528/76181548-71602280-61fc-11ea-85ef-2b8d9f63b530.png)
![wait-mutex](https://user-images.githubusercontent.com/19700528/76181554-76bd6d00-61fc-11ea-96ed-500ea58720da.png)
![get-duration](https://user-images.githubusercontent.com/19700528/76181566-7cb34e00-61fc-11ea-9f67-56338c49c5e0.png)
![write-duration](https://user-images.githubusercontent.com/19700528/76181571-850b8900-61fc-11ea-9fdf-fe49fdc93d92.png)
![compaction-rate](https://user-images.githubusercontent.com/19700528/76181575-8b016a00-61fc-11ea-98a2-a3474f71b1f9.png)
![num-files](https://user-images.githubusercontent.com/19700528/76181583-92c10e80-61fc-11ea-99ef-d609a7b043d3.png)

### Steps to reproduce the behavior

- Set `target_file_size` = 8MB, and insert data to let the number of the total SST files reach about 40K
- Run read and write workload

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.