Suggestion on RocksDB Configuration

Hello,

I am looking for suggestions on my RocksDB configuration. Our use case is to load 100GB of key-value pairs into rocksdb and at run time only serve the key-value pairs in the db. Key is 32 bytes and value is 1.6 KB in size.

What we have right now is we used hadoop to generate a 100GB sst file using SstFileWriter api and save it off in S3. Each new server that comes up ingests the file using: db.ingestExternalFile(..). We use a i3.large machine (15.25 GiB | 2 vCPUs | 475 GiB NVMe SSD). P95 and avg response from rocksdb given the current configuration:

BlockSize = 2KB
FormatVersion = 4
Read-Write=100% read

is ~1ms but P99 and PMAX are pretty bad. We are looking for some way to reduce the PMAX response times which is ~10x P95.

Thanks.