kestra-io / kestra-io/plugin-serdes

IonToParquet compression level with Zstd

Open
#415 1 comment 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
1
Forks
20
Avg merge
1d 18h
Merged PRs (30d)
36

Description

### Feature description

In the serdes IonToParquet, we can choose the compression type (UNCOMPRESSED, SNAPPY, GZIP, ZSTD), but we can't choose the compression level for Zstd. I think without this, the default is 3, but it could be usefull to be able to change this value (ex: store cold result with 22 compression level)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the IonToParquet entry point and trace how the selected compression type is passed to the Parquet writer. Check how ZSTD currently uses its default level and where an additional compression-level option could be exposed. Done means callers can choose the Zstd level, including values such as 22, without changing the existing behavior for other compression types.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.