apache / apache/bookkeeper

[improve][server]Add entry digest in journal log file

Open
#4,659 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

Hi, I recently read BookKeeper journal source code, and I found that BookKeeper journal didn't use any digest to guarante journal data's integrity and consistency.
https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java#L811-L885
Due to the lack of digest validation, we don't know whether the journal log data is corrupted or not. So we can only just do replay scan until while loop breaks out or an IOException happens.
https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java#L822-L882
If journal data is corrupted(disk data corruption or modified by mistake), and scanning incorrect journal log data happens to produce some incorrect entries(entris that already exists in this bookie), the old entries will be replaced both in RocksDB and entryLog file, which may cause data loss.
https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieImpl.java#L598-L607
ZooKeeper txnLog will append a digest after a record. So I think BookKeeper also need to do that.
https://github.com/apache/zookeeper/blob/9e8296f678a9de8d9c30e3bc086a79c0cd39ef65/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnLog.java#L276-L327

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.