apache / apache/arrow-rs

Support advanced ZSTD compression parameters (window_log, etc.) in Parquet writer

Open
#9,807 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The Parquet writer currently accepts ZSTD compression via Compression::ZSTD(ZstdLevel), which only exposes the compression level (an i32). There's no way to pass advanced ZSTD parameters like window_log, search_log, or enable long-distance matching (LDM).

**Describe the solution you'd like**
Extend Compression::ZSTD (or add a new variant) to accept optional ZSTD advanced parameters. For example:

// Option A: new struct wrapping level + optional params
pub struct ZstdConfig {
pub level: ZstdLevel,
pub window_log: Option,
// future: search_log, long_distance_matching, etc.
}

// Option B: builder pattern
Compression::ZSTD(ZstdLevel::try_new(8)?.with_window_log(23))

**Describe alternatives you've considered**

One workaround is to re-compress outside the Parquet writer, which defeats the purpose of integrated column-level compression.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the Parquet writer's Compression::ZSTD handling and the existing ZstdLevel definition. Review how compression settings reach the writer, then determine how the requested advanced parameters should be represented; done means callers can provide parameters such as window_log and the writer applies them during column-level ZSTD compression.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.