apache / apache/iceberg-rust

BUG: failed to write compressed puffin footer

Open
#2,419 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

Hi team, [spec](https://iceberg.apache.org/puffin-spec/#footer-payload) requires puffin footer to be either uncompressed or LZ4 compressed. But currently we [don't implement](https://github.com/apache/iceberg-rust/blob/4f21e503afc5363c10fafac917f0febe136c464e/crates/iceberg/src/compression.rs#L121-L124) lz4, which means the compression flag is incorrect.

### To Reproduce

Example to reproduce
```rust
#[tokio::test]
async fn test_compress_footer_fails_because_lz4_is_unsupported() {
let temp_dir = TempDir::new().unwrap();
let file_io = FileIO::new_with_fs();
let path = temp_dir.path().join("compressed_footer.bin");
let output_file = file_io.new_output(path.to_str().unwrap()).unwrap();

// compress_footer=true sets the footer codec to LZ4, which is unimplemented.
let writer = PuffinWriter::new(&output_file, HashMap::new(), /*compress_footer=*/true)
.await
.unwrap();
writer.close().await.unwrap();
}
```
Error message
```sh
thread 'puffin::writer::tests::test_compress_footer_fails_because_lz4_is_unsupported' (400849) panicked at crates/iceberg/src/puffin/writer.rs:350:30:
called `Result::unwrap()` on an `Err` value: FeatureUnsupported => LZ4 compression is not supported currently

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
puffin::writer::tests::test_compress_footer_fails_because_lz4_is_unsupported
```

### Expected behavior

When compression option specified, footer payload should be compressed and written without issue.

### Willingness to contribute

None

Contributor guide

Open the contributing guide

Research direction

Start with the compression handling in crates/iceberg/src/compression.rs, then follow the footer-writing path in crates/iceberg/src/puffin/writer.rs. Run the reproduced Puffin writer test with compress_footer=true and verify that the footer is compressed and written without the unsupported-LZ4 error, with the compression flag matching the payload.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.