apache / apache/parquet-java

Pig and Hive cannot read repeated groups written with parquet-protobuf

Open
#1,410 1 comment 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

parquet-hive and parquet-pig make assumptions about list schemas that are not compatible with the more compact schemas generated by parquet-protobuf. This bug was discussed in more detail on https://github.com/Parquet/parquet-mr/issues/354

**Reporter**: [Nathan Howell](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=NathanHowell)

**Note**: *This issue was originally created as [PARQUET-14](https://issues.apache.org/jira/browse/PARQUET-14). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the list-schema handling in parquet-hive and parquet-pig, then use the discussion in issue 354 to understand how parquet-protobuf produces compact repeated-group schemas. Done means both readers accept those schemas without breaking existing list-schema handling; the payload names no specific files or tests, so those entry points must be located first.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.