Rocksdb instance throws Checksum Mismatch exception frequently with high workload in directio mode
- Dominant language
- Java
- Stars
- 2k
- Forks
- 976
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 7
Description
**BUG REPORT**
***Describe the bug***
Rocksdb instance throws Checksum Mismatch exception frequently with high workload in directio mode
***To Reproduce***
Steps to reproduce the behavior:
1. Run with high workload with openmessaging-benchmark(2000MB/s)
- 3 bookies
- 3 benchmark-worker
- 3 replica
- 3 writeQuorumSize
- 2 ackQuorumSize
2. See error
> Oct 20 18:34:46 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:46 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:47 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:47 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:48 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:48 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:49 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
***Expected behavior***
Caused by: org.rocksdb.RocksDBException: block checksum mismatch
***Screenshots***
If applicable, add screenshots to help explain your problem.

***Additional context***
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.