[Feature Request]: Parquet Output Transform - Show the maximum row width in bytes on the dialog
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Figuring out the size of a parquet file output is dependent on several factors. One of the more important factors is the width of the row in bytes. When the user gets fields to output, we know the metadata of those fields and either the specific field width/size or the underlying data type's maximum width.
I suggest putting a label and calculating this size and showing it to the user on the UI so they can decide how to adjust their row splits or adjust the other fields like row group size (also in bytes), etc.
This will help them with calculating the optimal file size for letting query engines read their files on object storage.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start by locating the Parquet Output Transform dialog and the code that handles selected output-field metadata. Determine how field widths or maximum widths are represented, then verify that the dialog displays the calculated maximum row width in bytes so users can compare it with row-splitting and row-group settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100