facebook / facebook/rocksdb

Performance regression compare 5.18.3 with 5.17.2

Open
#5,774 15 comments 0 reactions 1 assignee Claimed by @mrambacher View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

### Expected behavior
RocksDB-5.18.3 has better or at least equivalent performance compared with RocksDB-5.17.2

Apache Flink maintains its own rocksDB due to some necessary feature which [not accepted](https://github.com/facebook/rocksdb/pull/4463) by rocksDB community. Current RocksDB version is 5.17.2 and we plan to upgrade to 5.18.3 recently to use write buffer manager to control the native memory usage. We expect RocksDB-5.18.3 should behave better or at least equivalent performance compared with RocksDB-5.17.2.

### Actual behavior
Flink has its own [benchmark](https://github.com/dataArtisans/flink-benchmarks) to verify the performance of rocksDB state actions. We firstly found the performance regression on our own built frocksDB, although not too much (below 9%). To verify where the performance regression happened. We remove the Flink part of code using our customized feature and re-run the benchmark with official release RocksDB, however the regression still exists as the same.

Below is the experimental result:

| State operation | with RocksDB 5.17.2 | RocksDB 5.18.3 | regression |
| :------------- | :----------: | -----------: |-----------: |
| ListStateBenchmark.listAdd | 481.528 ± 5.320| 459.345 ± 4.371| 5% |
| ListStateBenchmark.listAddAll | 268.906 ± 3.992| 257.455 ± 3.033| 4% |
| ListStateBenchmark.listGet | 132.305 ± 1.002| 127.359 ± 2.253| 4% |
| ListStateBenchmark.listGetAndIterate | 131.320 ± 1.283| 128.804 ± 0.730| 2% |
| ListStateBenchmark.listUpdate | 485.186 ± 4.320| 462.605 ± 4.834| 5% |
| MapStateBenchmark.mapAdd | 417.429 ± 22.103| 391.960 ± 18.057| 6% |
| MapStateBenchmark.mapContains | 50.343 ± 0.174| 49.693 ± 0.114| 1% |
| MapStateBenchmark.mapEntries | 257.778 ± 9.273| 242.297 ± 9.154| 6% |
| MapStateBenchmark.mapGet | 49.723 ± 0.296| 49.257 ± 0.156| 1% |
| MapStateBenchmark.mapIterator | 348.866 ± 9.187| 331.473 ± 8.750| 5% |
| MapStateBenchmark.mapKeys | 349.904 ± 9.032| 334.288 ± 9.025| 4% |
| MapStateBenchmark.mapPutAll | 96.494 ± 7.024| 92.204 ± 8.478| 4% |
| MapStateBenchmark.mapRemove | 414.616 ± 26.312| 385.804 ± 28.014| 7% |
| MapStateBenchmark.mapUpdate | 409.037 ± 20.451| 384.314 ± 18.444| 6% |
| MapStateBenchmark.mapValues | 351.461 ± 9.121| 328.171 ± 8.600| 7% |
| ValueStateBenchmark.valueAdd | 416.528 ± 19.593| 394.626 ± 18.759| 5% |
| ValueStateBenchmark.valueGet | 620.408 ± 44.076| 561.931 ± 39.661| 9% |
| ValueStateBenchmark.valueUpdate | 416.214 ± 32.661| 400.443 ± 18.051| 4% |

### Steps to reproduce the behavior

1. mvn install Flink from my [branch](https://github.com/Myasuka/flink/tree/release-1.9-no-rocksdb-ttl) which removes the usage of Flink-customized RocksDB operations.
1. Modify the `pom.xml` of [flink benchmark](https://github.com/dataArtisans/flink-benchmarks) to change the rocksdb version.
1. execute the [flink benchmark](https://github.com/dataArtisans/flink-benchmarks) with command
~~~ shell
mvn -Dflink.version=1.5.0 clean package exec:exec \
-Dexec.executable=java -Dexec.args="-jar target/benchmarks.jar -rf csv org.apache.flink.state.benchmark.*"
~~~

Moreover, when we build RocksDB-5.18.3 from official tag, we also found two bugs blocking the release phase, and recorded it in [rocksdb dev group](https://www.facebook.com/groups/rocksdb.dev).
Apart from above regression result, we have two questions:
1. what is the standard release steps to release rocksDB? Would anything else left behind in [rocksDB-5.18.3](https://github.com/facebook/rocksdb/tree/v5.18.3)?
1. what performance test would rocksDB take before release? We could repeat this to ensure no performance regression.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.