Seeking Advice for Optimizing RocksDB Performance

RocksDB 8.5.4

Hi, I'm using RocksDB for a read-heavy use case. The DB is built in one central server, and the files are distributed to various replicas to bootstrap data.

Here are some key metrics:

Average row size: 5KB
Block size: 32KB
Database size on disk: 6GB
Number of records: 20M
Block cache size: 10GB
Average read latency: 200ms
Peak read QPS: 68K
Multiget batch size: 500
Total read threads: 350

And, no compactions or writes happening at the time of reads.

I'm eager to explore if there's potential to further enhance RocksDB's performance. Previously, I managed to improve data block cache hits from 50% to over 90% through various adjustments. Consequently, disk-related bottlenecks have become rare.

Now, I suspect that any remaining read latency might be linked to block cache contention or possibly another overlooked factor. Is there room for further optimization, or have I reached a performance ceiling?

I'd greatly appreciate any insights or suggestions on this matter.

Thank you!