apache / apache/hudi

Not use magic number of next block to determine current log block

Open
#15,081 0 comments 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
from-jira priority:blocker priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

HoodieLogFileReader use magic number of next log block to determine if current log block is corrupted. However, when next block has a corrupted magic number, we will abandon current block, which leads to data loss.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3694
- Type: Bug
- Fix version(s):
- 1.1.0

---

## Comments

30/Apr/23 18:15;shivnarayan;This might potentially be an issue only for hdfs like storage schemes where appends are feasible. ;;;

---

02/May/23 11:17;codope;Unlike S3 or GCS, writes are not transactional in HDFS. So, in case of S3 and GCS we write out each log file and the atomicity is assured by the underlying filesystem i.e. either complete file is written or not written at all. While in HDFS, there can be partially written log blocks, and hence it is possible that magic header in the next log block is corrupt. 

One possible solution is to throw {{CorruptedLogFileException}} from {{HoodieLogFileReader#isBlockCorrupted}} if the magic header is corrupt and {{{}StorageSchemes.isWriteTransaction{}}}.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.