apache / apache/iceberg

Parquet per column compression

Open
#16,090 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

Enable parquet per column compression based on https://github.com/apache/parquet-java/pull/3526 and https://github.com/apache/parquet-java/pull/3396

Example:
Iceberg table can be created with
```
spark.sql(
"CREATE TABLE local.default.test_per_col ("
+ " int_col int,"
+ " string_col string"
+ ") USING iceberg"
+ " TBLPROPERTIES ("
+ " 'write.parquet.compression-codec' = 'zstd',"
+ " 'write.parquet.compression-codec.column.int_col' = 'snappy'"
+ ")");
```

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the referenced parquet-java pull requests and locating how Iceberg handles the existing write.parquet.compression-codec table property. Trace the example properties for a column-specific codec, then add coverage for creating and writing the shown table; done means each column uses its configured codec while the table-wide setting remains the default.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.