Fail the reads from log record reader for any corruptness
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In GCS and S3 like systems, where partial writes failures are not possible, we are planning to fail (fail fast) the reads from log files if hudi deduces any corrupt data rather than silently skip and move on. Since writes are atomic and only effective if the entire data is fully written, this should be fine with GCS and S3 schemes.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6183
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating Hudi's log record reader and the handling of corrupt records, then review the linked HUDI-6183 requirements. Done means reads from log files fail fast when corrupt data is detected for GCS, S3, and similar atomic-write schemes instead of silently skipping it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, google-cloud, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100