apache / apache/parquet-java

Parquet file containing arrays, written by Parquet-MR, cannot be read again by Parquet-MR

Open
#2,631 18 comments 0 reactions 0 assignees View on GitHub
Component: Avro Component: Parquet Priority: Blocker Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

In the attached files, there is one original file, and one written modified file that results after reading the original file and writing it back with Parquet-MR, with a few values modified. The schema should not be modified, since the schema of the input file is used as the schema to write the output file. However, the output file has a slightly modified schema that then cannot be read back the same way again with Parquet-MR, resulting in the exception message:  java.lang.ClassCastException: optional binary element (STRING) is not a group

My guess is that the issue lies in the Avro schema conversion.

The Parquet files attached have some arrays and some nested fields.

**Environment**: Windows 10
**Reporter**: [Devon Kozenieski](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=devonk)
#### Related issues:
- [parquet schema conflict: optional binary (UTF8) is not a group](https://issues.apache.org/jira/browse/HUDI-864) (is depended upon by)
#### Original Issue Attachments:
- [modified.parquet](https://issues.apache.org/jira/secure/attachment/13030885/modified.parquet)
- [original.parquet](https://issues.apache.org/jira/secure/attachment/13030884/original.parquet)
- [parquet-diff.png](https://issues.apache.org/jira/secure/attachment/13042124/parquet-diff.png)

**Note**: *This issue was originally created as [PARQUET-2069](https://issues.apache.org/jira/browse/PARQUET-2069). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the attached original.parquet and modified.parquet files, then trace the Avro schema conversion used when Parquet-MR reads and rewrites nested arrays. Reproduce the ClassCastException while reading the rewritten file; done means the schema remains compatible and the output can be read again by Parquet-MR.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.