apache / apache/opendal

new feature: Checksum

Open
#5,549 9 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Checksum is extremely important for storage services. We currently support some checksum features in OpenDAL, but they are very limited: we only support setting [checksum_algorithm](https://docs.rs/opendal/0.51.1/opendal/services/struct.S3.html#method.checksum_algorithm) for S3.

I hope we can introduce this feature across all services, supporting newer and faster checksum algorithms such as `crc64-nvme` or `crc64-ecma`.

### Problem and Solution

- Add `checksum_algorithm` for all supported services.
- Add additional `checksum_algorithm` options for S3, such as `crc64-nvme`.
- We can also add some slow checksums like sha1 and md5

### Additional Context

- Perhaps we should also expose this at the metadata level.
- Should we verify the checksum during the reading process?
- Do we need to introduce a new error type for this?
- We should re-consider the current design.
- Is it a good idea to add at service level?
- Is it better to have a `ChecksumLayer` for this?

### Are you willing to contribute to the development of this feature?

- [ ] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start with the existing S3 checksum_algorithm entry point and compare how checksum behavior is represented across supported services. Resolve whether configuration belongs at service level or in a ChecksumLayer, whether metadata and read-time verification are required, and whether a new error type is needed. Done means the design is settled and checksum algorithms are supported consistently across services.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.