apache / apache/hop

[Bug]: Parquet Output - Parquet Footer Size.

Open
#2,338 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 33m
Merged PRs (30d)
204

Description

### Apache Hop version?

2.3

### Java version?

11

### Operating system

Windows

### What happened?

Migration: https://issues.apache.org/jira/browse/HOP-4271

Is there any way that we can put on the parquet output dialog box the calculated footer size before running the transform?
Could we make it more clear what variables play into the size of the footer?

Situation:

Hop is correctly generating Parquet v2 files, but certain systems like Dremio try to read footer information from the end of the parquet files.  If the size of the entire footer is greater than 16MB, then the system throws the error below.

Caused by: java.io.IOException: Footer size of /datalake-bucket/PickedUpAccountingReportParquet/PickedUpAccountingReport-20221003-132903-00-0001.parquet.snappy is 92092708. Max supported footer size is 16777216

From experimentation, there are adjustments to when we split a file that seem to influence the size of the footer created.

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start with the Parquet output dialog and the file-splitting settings described in the report; reproduce the 92,092,708-byte footer against the stated Hop 2.3, Java 11, and Windows setup. Determine which footer-size inputs can be calculated before execution, then verify the estimate against generated files and make the relevant variables clear in the dialog.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.