apache / apache/iceberg

Add a property to configure the checksum algorithm in S3FileIO

Open
#17,178 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

S3FileIO currently has no way to choose which checksum algorithm the AWS SDK uses for data integrity protection on S3 writes. Since AWS SDK 2.30.0, the SDK calculates a CRC32 checksum by default for requests that support one, and the only related knobs are the legacy Content-MD5 (`s3.checksum-enabled`) and, once #17177 lands, the calculation/validation policies (`s3.request-checksum-calculation` / `s3.response-checksum-validation`) — none of which select the algorithm itself.

I propose adding a new property:

- `s3.checksum-algorithm`: selects the checksum algorithm the SDK uses for S3FileIO writes. When unset, the AWS SDK default (CRC32) is used, so there is no behavior change.

S3 supports CRC32, CRC32C, CRC64NVME, SHA-1, and SHA-256, and added MD5, SHA-512, XXHash3, XXHash64, and XXHash128 in April 2026 (available in the AWS SDK since 2.42.x, already covered by the SDK version Iceberg uses today). Use cases include:

- CRC64NVME for efficient full-object integrity checks (AWS-recommended for multipart uploads, since it composes across parts)
- SHA-256 where compliance requirements mandate a cryptographic hash
- Matching the algorithm an organization already standardizes on for object verification

The property would be applied per request in `S3OutputStream` (`PutObjectRequest`, `CreateMultipartUploadRequest`, `UploadPartRequest`), following how other write options are applied there.

One interaction to document: setting a request-level checksum algorithm causes the SDK to calculate checksums even when `s3.request-checksum-calculation` is `when_required`, so an explicit algorithm effectively opts back in to checksum calculation.

Related: #14439, #17177

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start in S3OutputStream and review how existing write options are applied to PutObjectRequest, CreateMultipartUploadRequest, and UploadPartRequest. Read related issues #14439 and #17177 for the surrounding checksum properties and interaction. Done means adding s3.checksum-algorithm while preserving the AWS SDK default when it is unset and documenting its effect on request checksum calculation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.