apache / apache/arrow-rs-object-store
[object_store] Support adding MD5 checksum headers to validate object integrity
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'd like verify the integrity of uploaded objects (using some kind of checksum) across all three cloud providers.
Currently, the S3 implementation allows setting an AmazonS3 configuration value that attaches a `x-amz-checksum-sha256` header to PUT requests against the store. However:
- This is only available for AWS (and SHA256).
- AWS supports other checksum algorithms (MD5, SHA1, CRC32, CRC32C)
- Azure supports MD5 and CRC64
- GCP supports MD5 and CRC32C
- This requires another pass to calculate the checksum value, which the user of this library may already have computed in another context.
**Describe the solution you'd like**
I'm proposing that we add a `Checksum` attribute which specifies a `ChecksumAlgorithm` enum. The value for this attribute would be the base64-encoded checksum value. For now, MD5 can be the only supported checksum algorithm (which all three cloud providers support via the `Content-MD5` header). The value for this algorithm is a base64-encoded 128-bit digest.
```
pub enum Attribute {
...
/// Provides a checksum used to verify object data integrity
Checksum(ChecksumAlgorithm),
}
pub enum ChecksumAlgorithm {
MD5,
}
```
**Describe alternatives you've considered**
I considered implementing this for more checksum algorithms, but I'm starting with MD5 because it's the only one supported by all three cloud providers. In the future, we could extend this to support additional checksum algorithms (e.g. CRC32C). However:
- stores that do not support a particular error would need to return an error
- stores would ideally be able to "report" their supported checksum algorithms (for usability reasons)
- (AWS) we'd need to figure out how a SHA-256 checksum provided via attributes interacts with the one provided via config
I considered calling the attribute `ContentMD5` but decided to make it a bit more generic to support additional checksums in the future.
**Additional context**
I can put up a PR for this issue.
Contributor guide
Research direction
Trace the existing S3 checksum configuration and PUT handling, then compare how uploads are implemented for AWS, Azure, and GCP. Define how the proposed Attribute and ChecksumAlgorithm fit those paths; done means MD5 can be supplied as a base64-encoded value through Content-MD5 for all three providers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure, gcp, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100