[Bug]: Parquet Output - Parquet Footer Size.
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 33m
- Merged PRs (30d)
- 204
Description
### Apache Hop version?
2.3
### Java version?
11
### Operating system
Windows
### What happened?
Migration: https://issues.apache.org/jira/browse/HOP-4271
Is there any way that we can put on the parquet output dialog box the calculated footer size before running the transform?
Could we make it more clear what variables play into the size of the footer?
Situation:
Hop is correctly generating Parquet v2 files, but certain systems like Dremio try to read footer information from the end of the parquet files. If the size of the entire footer is greater than 16MB, then the system throws the error below.
Caused by: java.io.IOException: Footer size of /datalake-bucket/PickedUpAccountingReportParquet/PickedUpAccountingReport-20221003-132903-00-0001.parquet.snappy is 92092708. Max supported footer size is 16777216
From experimentation, there are adjustments to when we split a file that seem to influence the size of the footer created.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start with the Parquet output dialog and the file-splitting settings described in the report; reproduce the 92,092,708-byte footer against the stated Hop 2.3, Java 11, and Windows setup. Determine which footer-size inputs can be calculated before execution, then verify the estimate against generated files and make the relevant variables clear in the dialog.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100