apache / apache/hop

[Feature Request]: Parquet Output Transform - Show the maximum row width in bytes on the dialog

Open
#2,903 1 comment 0 reactions 0 assignees View on GitHub
awaiting triage P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### What would you like to happen?

Figuring out the size of a parquet file output is dependent on several factors. One of the more important factors is the width of the row in bytes. When the user gets fields to output, we know the metadata of those fields and either the specific field width/size or the underlying data type's maximum width.

I suggest putting a label and calculating this size and showing it to the user on the UI so they can decide how to adjust their row splits or adjust the other fields like row group size (also in bytes), etc.

This will help them with calculating the optimal file size for letting query engines read their files on object storage.

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start by locating the Parquet Output Transform dialog and the code that handles selected output-field metadata. Determine how field widths or maximum widths are represented, then verify that the dialog displays the calculated maximum row width in bytes so users can compare it with row-splitting and row-group settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.