BUG: failed to write compressed puffin footer
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### Apache Iceberg Rust version
None
### Describe the bug
Hi team, [spec](https://iceberg.apache.org/puffin-spec/#footer-payload) requires puffin footer to be either uncompressed or LZ4 compressed. But currently we [don't implement](https://github.com/apache/iceberg-rust/blob/4f21e503afc5363c10fafac917f0febe136c464e/crates/iceberg/src/compression.rs#L121-L124) lz4, which means the compression flag is incorrect.
### To Reproduce
Example to reproduce
```rust
#[tokio::test]
async fn test_compress_footer_fails_because_lz4_is_unsupported() {
let temp_dir = TempDir::new().unwrap();
let file_io = FileIO::new_with_fs();
let path = temp_dir.path().join("compressed_footer.bin");
let output_file = file_io.new_output(path.to_str().unwrap()).unwrap();
// compress_footer=true sets the footer codec to LZ4, which is unimplemented.
let writer = PuffinWriter::new(&output_file, HashMap::new(), /*compress_footer=*/true)
.await
.unwrap();
writer.close().await.unwrap();
}
```
Error message
```sh
thread 'puffin::writer::tests::test_compress_footer_fails_because_lz4_is_unsupported' (400849) panicked at crates/iceberg/src/puffin/writer.rs:350:30:
called `Result::unwrap()` on an `Err` value: FeatureUnsupported => LZ4 compression is not supported currently
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
puffin::writer::tests::test_compress_footer_fails_because_lz4_is_unsupported
```
### Expected behavior
When compression option specified, footer payload should be compressed and written without issue.
### Willingness to contribute
None
Contributor guide
Research direction
Start with the compression handling in crates/iceberg/src/compression.rs, then follow the footer-writing path in crates/iceberg/src/puffin/writer.rs. Run the reproduced Puffin writer test with compress_footer=true and verify that the footer is compressed and written without the unsupported-LZ4 error, with the compression flag matching the payload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100