High latency at low QPS on reloading DB
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
using rockdb v8.3
We have some production serving nodes that host 1TB data . Periodically we download new set of SST files (fully compacted on a different indexing node). On Serving node , we open new DB instance pointing to new compacted SST files
Observation : During opening new DB , we observe the serving DB latency spikes up to as high has 2 sec P99_9 (typically this is in order of < 20ms) . Whats puzzling is this happens when serving nodes have low QPS , we do not observe this latency spikes when we serve a mich higher qps (for example when QPS is increased by a factor of 5x) .
2 relevant configurations we tried but didnt help
max_open_files = -1
max_file_opening_threads=16
all rocksdb configuration were same .
Any ideas or hypothesis why lower traffic observes such issues .
Any params we should tune in particular , or if you can elaborate a bit on what exactly happens on opening a DB
in particular : any data strucutres that need to be warmed up ? any os page cache pollution because i read that on opening a DB need to read some segment of each SST files. into memory ?
Contributor guide
Research direction
No source file or test is named. Begin at the RocksDB v8.3 DB-opening and reload path, reproduce the P99.9 latency spike with the stated SST-file workload at low and higher QPS, and document the responsible initialization or cache behavior plus any validated parameter change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100