apache / apache/parquet-java

ParquetInputSplit end calculation bug

Open
#1,750 0 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The calculation for end of a split using the file metadata is broken by PARQUET-108. The calculation was updated to use the requested schema so that the end of a block would be the end of the last projected column. But [the end logic](https://github.com/apache/incubator-parquet-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/ParquetInputSplit.java#L94) actually calculates the total number of bytes that are selected.

The end of a split is only used to select row groups when a block has no row group offsets, which doesn't happen when the constructor that uses the broken method is called. However, this should still be removed.

After 1.6.0, I want to move Hive to pass FileSplits directly rather than wrapping them in ParquetInputSplit. The internal reader code can handle mapping row groups to splits because it needs to for PARQUET-84.

**Reporter**: [Ryan Blue](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=rdblue) / @rdblue

**Note**: *This issue was originally created as [PARQUET-207](https://issues.apache.org/jira/browse/PARQUET-207). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with parquet-hadoop/src/main/java/parquet/hadoop/ParquetInputSplit.java at the linked end-logic location and trace the constructor path that uses it. Check how the end value is used when row-group offsets are absent. Done means removing the broken calculation without changing the supported split and row-group behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.