[Parquet] statistics_truncate_length as column properties?
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently, the Parquet Writer has `WriterProperties`. In writer properties, here is also `column_properties` and `default_column_properties`, which means the "specific column properties"
And `column_index_truncate_length`, `statistics_truncate_length` is regarded as global properties.
Here, we can disable statistics for a column. Since we have `ColumnProperties.statistics_enabled`. However, we might requires:
1. For primary key, we understand it would not be too long, so we don't want to truncate it.
2. Other columns, we think we can truncate the lengths
**Describe the solution you'd like**
Change the `column_index_truncate_length`, `statistics_truncate_length` to column properties
**Describe alternatives you've considered**
I don't know
**Additional context**
No
Contributor guide
Research direction
Start by tracing the Parquet Writer's WriterProperties, column_properties, and default_column_properties handling. Compare how column_index_truncate_length and statistics_truncate_length are currently applied with ColumnProperties.statistics_enabled. Done means these truncation settings can be configured per column while preserving the existing global behavior where appropriate, with tests covering differing limits for primary-key and other columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100