Support advanced ZSTD compression parameters (window_log, etc.) in Parquet writer
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The Parquet writer currently accepts ZSTD compression via Compression::ZSTD(ZstdLevel), which only exposes the compression level (an i32). There's no way to pass advanced ZSTD parameters like window_log, search_log, or enable long-distance matching (LDM).
**Describe the solution you'd like**
Extend Compression::ZSTD (or add a new variant) to accept optional ZSTD advanced parameters. For example:
// Option A: new struct wrapping level + optional params
pub struct ZstdConfig {
pub level: ZstdLevel,
pub window_log: Option,
// future: search_log, long_distance_matching, etc.
}
// Option B: builder pattern
Compression::ZSTD(ZstdLevel::try_new(8)?.with_window_log(23))
**Describe alternatives you've considered**
One workaround is to re-compress outside the Parquet writer, which defeats the purpose of integrated column-level compression.
**Additional context**
Contributor guide
Research direction
Start by locating the Parquet writer's Compression::ZSTD handling and the existing ZstdLevel definition. Review how compression settings reach the writer, then determine how the requested advanced parameters should be represented; done means callers can provide parameters such as window_log and the writer applies them during column-level ZSTD compression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100