Performance regression compare 5.18.3 with 5.17.2
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
### Expected behavior
RocksDB-5.18.3 has better or at least equivalent performance compared with RocksDB-5.17.2
Apache Flink maintains its own rocksDB due to some necessary feature which [not accepted](https://github.com/facebook/rocksdb/pull/4463) by rocksDB community. Current RocksDB version is 5.17.2 and we plan to upgrade to 5.18.3 recently to use write buffer manager to control the native memory usage. We expect RocksDB-5.18.3 should behave better or at least equivalent performance compared with RocksDB-5.17.2.
### Actual behavior
Flink has its own [benchmark](https://github.com/dataArtisans/flink-benchmarks) to verify the performance of rocksDB state actions. We firstly found the performance regression on our own built frocksDB, although not too much (below 9%). To verify where the performance regression happened. We remove the Flink part of code using our customized feature and re-run the benchmark with official release RocksDB, however the regression still exists as the same.
Below is the experimental result:
| State operation | with RocksDB 5.17.2 | RocksDB 5.18.3 | regression |
| :------------- | :----------: | -----------: |-----------: |
| ListStateBenchmark.listAdd | 481.528 ± 5.320| 459.345 ± 4.371| 5% |
| ListStateBenchmark.listAddAll | 268.906 ± 3.992| 257.455 ± 3.033| 4% |
| ListStateBenchmark.listGet | 132.305 ± 1.002| 127.359 ± 2.253| 4% |
| ListStateBenchmark.listGetAndIterate | 131.320 ± 1.283| 128.804 ± 0.730| 2% |
| ListStateBenchmark.listUpdate | 485.186 ± 4.320| 462.605 ± 4.834| 5% |
| MapStateBenchmark.mapAdd | 417.429 ± 22.103| 391.960 ± 18.057| 6% |
| MapStateBenchmark.mapContains | 50.343 ± 0.174| 49.693 ± 0.114| 1% |
| MapStateBenchmark.mapEntries | 257.778 ± 9.273| 242.297 ± 9.154| 6% |
| MapStateBenchmark.mapGet | 49.723 ± 0.296| 49.257 ± 0.156| 1% |
| MapStateBenchmark.mapIterator | 348.866 ± 9.187| 331.473 ± 8.750| 5% |
| MapStateBenchmark.mapKeys | 349.904 ± 9.032| 334.288 ± 9.025| 4% |
| MapStateBenchmark.mapPutAll | 96.494 ± 7.024| 92.204 ± 8.478| 4% |
| MapStateBenchmark.mapRemove | 414.616 ± 26.312| 385.804 ± 28.014| 7% |
| MapStateBenchmark.mapUpdate | 409.037 ± 20.451| 384.314 ± 18.444| 6% |
| MapStateBenchmark.mapValues | 351.461 ± 9.121| 328.171 ± 8.600| 7% |
| ValueStateBenchmark.valueAdd | 416.528 ± 19.593| 394.626 ± 18.759| 5% |
| ValueStateBenchmark.valueGet | 620.408 ± 44.076| 561.931 ± 39.661| 9% |
| ValueStateBenchmark.valueUpdate | 416.214 ± 32.661| 400.443 ± 18.051| 4% |
### Steps to reproduce the behavior
1. mvn install Flink from my [branch](https://github.com/Myasuka/flink/tree/release-1.9-no-rocksdb-ttl) which removes the usage of Flink-customized RocksDB operations.
1. Modify the `pom.xml` of [flink benchmark](https://github.com/dataArtisans/flink-benchmarks) to change the rocksdb version.
1. execute the [flink benchmark](https://github.com/dataArtisans/flink-benchmarks) with command
~~~ shell
mvn -Dflink.version=1.5.0 clean package exec:exec \
-Dexec.executable=java -Dexec.args="-jar target/benchmarks.jar -rf csv org.apache.flink.state.benchmark.*"
~~~
Moreover, when we build RocksDB-5.18.3 from official tag, we also found two bugs blocking the release phase, and recorded it in [rocksdb dev group](https://www.facebook.com/groups/rocksdb.dev).
Apart from above regression result, we have two questions:
1. what is the standard release steps to release rocksDB? Would anything else left behind in [rocksDB-5.18.3](https://github.com/facebook/rocksdb/tree/v5.18.3)?
1. what performance test would rocksDB take before release? We could repeat this to ensure no performance regression.
Contributor guide
Assessment
This issue has not been assessed yet.