new feature: Add more checksum support for S3
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
`crc64-nvme` is the new crc algo that supported by s3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
### Problem and Solution
- [ ] CRC64-NVME (Recommended)
- `crc64-nvme` is the default algo used by s3. Adding it in opendal can make sure the data we upload is correct.
- [ ] CRC-32
- [x] CRC-32C
- [ ] SHA-1
- [ ] SHA-256
- [ ] Also, `content-md5`.
### Additional Context
_No response_
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start by reading the linked AWS S3 object-integrity documentation, then locate OpenDAL's S3 service implementation and its existing CRC-32C checksum handling. Determine how the remaining requested algorithms—CRC64-NVME, CRC-32, SHA-1, SHA-256, and content-MD5—fit the current integration, and verify the supported checksum behavior against S3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100