Not use magic number of next block to determine current log block
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
HoodieLogFileReader use magic number of next log block to determine if current log block is corrupted. However, when next block has a corrupted magic number, we will abandon current block, which leads to data loss.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3694
- Type: Bug
- Fix version(s):
- 1.1.0
---
## Comments
30/Apr/23 18:15;shivnarayan;This might potentially be an issue only for hdfs like storage schemes where appends are feasible. ;;;
---
02/May/23 11:17;codope;Unlike S3 or GCS, writes are not transactional in HDFS. So, in case of S3 and GCS we write out each log file and the atomicity is assured by the underlying filesystem i.e. either complete file is written or not written at all. While in HDFS, there can be partially written log blocks, and hence it is possible that magic header in the next log block is corrupt.
One possible solution is to throw {{CorruptedLogFileException}} from {{HoodieLogFileReader#isBlockCorrupted}} if the magic header is corrupt and {{{}StorageSchemes.isWriteTransaction{}}}.;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.