apache / apache/iceberg-rust

[Bug] iceberg-rust does not respect compression settings for metadata & avro

Open
#1,731 3 comments 2 reactions 0 assignees View on GitHub
bug not-stale
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

0.6

### Describe the bug

I'm using the Nessie catalog with the following settings:
```
- nessie.catalog.iceberg-config-defaults.write.avro.compression-level=3
- nessie.catalog.iceberg-config-defaults.write.avro.compression-codec=zstd
- nessie.catalog.iceberg-config-defaults.write.metadata.compression-codec=gzip
```
These settings are passed to the clients, but `iceberg-rust` does not have code that respects them.
As a result, it keeps producing uncompressed metadata and Avro files.

### To Reproduce

_No response_

### Expected behavior

_No response_

### Willingness to contribute

None

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the Nessie configuration keys write.avro.compression-level, write.avro.compression-codec, and write.metadata.compression-codec reach the metadata and Avro writers. Confirm the affected output currently lacks compression, then verify that the configured codecs and level are respected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.