apache / apache/bookkeeper

Rocksdb instance throws Checksum Mismatch exception frequently with high workload in directio mode

Open
#3,567 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BUG REPORT**

***Describe the bug***

Rocksdb instance throws Checksum Mismatch exception frequently with high workload in directio mode

***To Reproduce***

Steps to reproduce the behavior:
1. Run with high workload with openmessaging-benchmark(2000MB/s)

- 3 bookies
- 3 benchmark-worker
- 3 replica
- 3 writeQuorumSize
- 2 ackQuorumSize

2. See error

> Oct 20 18:34:46 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:46 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:47 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:47 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:48 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:48 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:49 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026
Oct 20 18:34:50 test_host pulsar[6380]: Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 4048457838, computed = 3129443982, type = 2 in /cache2/storage/5/current/locations/000037.sst offset 0 size 29026

***Expected behavior***

Caused by: org.rocksdb.RocksDBException: block checksum mismatch

***Screenshots***

If applicable, add screenshots to help explain your problem.
image

***Additional context***

Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.