apache / apache/opendal

new feature: Add more checksum support for S3

Open
#5,550 4 comments 0 reactions 0 assignees View on GitHub
enhancement services/s3
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

`crc64-nvme` is the new crc algo that supported by s3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

### Problem and Solution

- [ ] CRC64-NVME (Recommended)
- `crc64-nvme` is the default algo used by s3. Adding it in opendal can make sure the data we upload is correct.
- [ ] CRC-32
- [x] CRC-32C
- [ ] SHA-1
- [ ] SHA-256
- [ ] Also, `content-md5`.

### Additional Context

_No response_

### Are you willing to contribute to the development of this feature?

- [ ] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked AWS S3 object-integrity documentation, then locate OpenDAL's S3 service implementation and its existing CRC-32C checksum handling. Determine how the remaining requested algorithms—CRC64-NVME, CRC-32, SHA-1, SHA-256, and content-MD5—fit the current integration, and verify the supported checksum behavior against S3.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.