Parquet per column compression
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
Enable parquet per column compression based on https://github.com/apache/parquet-java/pull/3526 and https://github.com/apache/parquet-java/pull/3396
Example:
Iceberg table can be created with
```
spark.sql(
"CREATE TABLE local.default.test_per_col ("
+ " int_col int,"
+ " string_col string"
+ ") USING iceberg"
+ " TBLPROPERTIES ("
+ " 'write.parquet.compression-codec' = 'zstd',"
+ " 'write.parquet.compression-codec.column.int_col' = 'snappy'"
+ ")");
```
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start by reviewing the referenced parquet-java pull requests and locating how Iceberg handles the existing write.parquet.compression-codec table property. Trace the example properties for a column-specific codec, then add coverage for creating and writing the shown table; done means each column uses its configured codec while the table-wide setting remains the default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100