apache / apache/parquet-java

Parquet to report number of rows affected in the row group when corrupted bytes occur

Open
#1,696 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

In case of hardware failure (disk, memory, etc), there might be corrupted bytes. That will result in ArrayIndexOutOfBoundException or/and data garbled.

Parquet should report number of rows being affected in case of such scenario.

**Reporter**: [Tongjie Chen](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=tongjie) / @tongjiechen

**Note**: *This issue was originally created as [PARQUET-147](https://issues.apache.org/jira/browse/PARQUET-147). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is named in the issue; begin by locating Parquet's row-group reading and corruption/error handling paths. Determine how corrupted bytes are detected and how affected rows can be counted, then add coverage showing that corruption reports the number of impacted rows without silently accepting garbled data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.