Parquet to report number of rows affected in the row group when corrupted bytes occur
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
In case of hardware failure (disk, memory, etc), there might be corrupted bytes. That will result in ArrayIndexOutOfBoundException or/and data garbled.
Parquet should report number of rows being affected in case of such scenario.
**Reporter**: [Tongjie Chen](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=tongjie) / @tongjiechen
**Note**: *This issue was originally created as [PARQUET-147](https://issues.apache.org/jira/browse/PARQUET-147). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file, test, or entry point is named in the issue; begin by locating Parquet's row-group reading and corruption/error handling paths. Determine how corrupted bytes are detected and how affected rows can be counted, then add coverage showing that corruption reports the number of impacted rows without silently accepting garbled data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100