facebook / facebook/rocksdb

High latency at low QPS on reloading DB

Open
#12,126 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

using rockdb v8.3

We have some production serving nodes that host 1TB data . Periodically we download new set of SST files (fully compacted on a different indexing node). On Serving node , we open new DB instance pointing to new compacted SST files

Observation : During opening new DB , we observe the serving DB latency spikes up to as high has 2 sec P99_9 (typically this is in order of < 20ms) . Whats puzzling is this happens when serving nodes have low QPS , we do not observe this latency spikes when we serve a mich higher qps (for example when QPS is increased by a factor of 5x) .
2 relevant configurations we tried but didnt help
max_open_files = -1
max_file_opening_threads=16

all rocksdb configuration were same .

Any ideas or hypothesis why lower traffic observes such issues .
Any params we should tune in particular , or if you can elaborate a bit on what exactly happens on opening a DB
in particular : any data strucutres that need to be warmed up ? any os page cache pollution because i read that on opening a DB need to read some segment of each SST files. into memory ?

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Begin at the RocksDB v8.3 DB-opening and reload path, reproduce the P99.9 latency spike with the stated SST-file workload at low and higher QPS, and document the responsible initialization or cache behavior plus any validated parameter change.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.