new feature: Checksum
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Checksum is extremely important for storage services. We currently support some checksum features in OpenDAL, but they are very limited: we only support setting [checksum_algorithm](https://docs.rs/opendal/0.51.1/opendal/services/struct.S3.html#method.checksum_algorithm) for S3.
I hope we can introduce this feature across all services, supporting newer and faster checksum algorithms such as `crc64-nvme` or `crc64-ecma`.
### Problem and Solution
- Add `checksum_algorithm` for all supported services.
- Add additional `checksum_algorithm` options for S3, such as `crc64-nvme`.
- We can also add some slow checksums like sha1 and md5
### Additional Context
- Perhaps we should also expose this at the metadata level.
- Should we verify the checksum during the reading process?
- Do we need to introduce a new error type for this?
- We should re-consider the current design.
- Is it a good idea to add at service level?
- Is it better to have a `ChecksumLayer` for this?
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start with the existing S3 checksum_algorithm entry point and compare how checksum behavior is represented across supported services. Resolve whether configuration belongs at service level or in a ChecksumLayer, whether metadata and read-time verification are required, and whether a new error type is needed. Done means the design is settled and checksum algorithms are supported consistently across services.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100