Merging two valid parquet files produces a corrupted result file in 1.12.1
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
This ticket relates to PARQUET-2027. In the previous ticket for two parquet files produced by 1.11.x merging was failing in 1.12.0. For 1.12.1 merging was fixed, i. e. it doesn't fail. But in the same time it results with a corrupted output file. The error:
Dictionary page must be before data page.
is thrown when one tries to read it. It comes from this https://github.com/apache/parquet-cpp/blob/master/src/parquet/arrow/record_reader.cc#L712.
I attached two example input files and the outcome of merging.
Reporter: Matthew M
Original Issue Attachments:
Note: This issue was originally created as PARQUET-2100. Please see the migration documentation for further details.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the merge with input_file1.parquet and input_file2.parquet using 1.12.1, then try reading output_file.parquet to confirm the dictionary-page error. Compare the failure with PARQUET-2027 and inspect the referenced parquet-cpp record_reader.cc line 712. Done means the merged output is readable without the reported corruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100