apache / apache/arrow-rs

[Parquet] Support configurable ALP vector lengths

Open
#10,645 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

### Is your feature request related to a problem or challenge?

The Parquet ALP specification permits `log_vector_size` values from 3 through 15, but arrow-rs currently writes only the recommended default size of 1024 values.

This prevents testing or selecting other vector sizes permitted by the format.

This depends on #9372 and should be worked on after that PR is merged.

### Describe the solution you would like

Add an ALP writer configuration for selecting any vector size permitted by the Parquet specification.

The existing default should remain 1024 values.

Contributor guide

Open the contributing guide

Research direction

Review dependency #9372 first, then inspect the existing ALP writer configuration and how it uses the default vector size. Confirm the writer accepts the Parquet-permitted log_vector_size range of 3 through 15, preserves the 1024-value default, and supports selecting another permitted size.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.