apache / apache/arrow-rs

[Parquet] statistics_truncate_length as column properties?

Open
#8,094 0 comments 1 reaction 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently, the Parquet Writer has `WriterProperties`. In writer properties, here is also `column_properties` and `default_column_properties`, which means the "specific column properties"

And `column_index_truncate_length`, `statistics_truncate_length` is regarded as global properties.

Here, we can disable statistics for a column. Since we have `ColumnProperties.statistics_enabled`. However, we might requires:
1. For primary key, we understand it would not be too long, so we don't want to truncate it.
2. Other columns, we think we can truncate the lengths

**Describe the solution you'd like**

Change the `column_index_truncate_length`, `statistics_truncate_length` to column properties

**Describe alternatives you've considered**

I don't know

**Additional context**

No

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Parquet Writer's WriterProperties, column_properties, and default_column_properties handling. Compare how column_index_truncate_length and statistics_truncate_length are currently applied with ColumnProperties.statistics_enabled. Done means these truncation settings can be configured per column while preserving the existing global behavior where appropriate, with tests covering differing limits for primary-key and other columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.