awslabs / awslabs/aws-sdk-rust

S3 PutObject: Set content length with a stream hyper Body for checksum

Open
#668 3 comments 1 reaction 0 assignees View on GitHub
feature-request p3
Dominant language
Rust
Stars
3.3k
Forks
290
Avg merge
1d 12h
Merged PRs (30d)
3

Description

### Describe the bug

Hi there!
We would want to set the `content_length` (as `size_hint`) with a streaming `hyper:Body` to upload an object to S3 with it's checksum.

### Expected Behavior

Upload the streaming body with it's checksum to S3.

### Current Behavior

Fail with a message like `Only request bodies with a known size can be checksum validated` coming from [there](https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/s3/src/http_body_checksum.rs#L25)

### Reproduction Steps

We are doing something like:

```

async upload(from: Box<(dyn io::AsyncRead + Unpin + Send + Sync + 'static)>, from_size_bytes: i64)
let body = hyper::Body::wrap_stream(tokio_util::io::ReaderStream::new(from));
let bytes_stream = ByteStream::new(SdkBody::from(body));

self.client
.put_object()
.bucket(self.bucket_name.clone())
.key(to.0.to_owned())
.set_metadata(metadata.map(|m| m.0))
.checksum_algorithm(ChecksumAlgorithm::Sha256)
.content_length(from_size_bytes) <--- a try to size_hint
.body(bytes_stream)
.send()
.await
.map_err(|e| {
metrics::inc_put_object_fail();
e
})?;
```

In Rusoto we used [new_with_size](https://docs.rs/rusoto_signature/latest/rusoto_signature/stream/struct.ByteStream.html#method.new_with_size) to specify the size_hint in the body stream.

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### Version

```text
├── aws-config v0.51.0
│ ├── aws-http v0.51.0
│ │ ├── aws-smithy-http v0.51.0
│ │ │ ├── aws-smithy-eventstream v0.51.0
│ │ │ │ ├── aws-smithy-types v0.51.0
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-types v0.51.0
│ │ │ ├── aws-smithy-async v0.51.0
│ │ │ ├── aws-smithy-client v0.51.0
│ │ │ │ ├── aws-smithy-async v0.51.0 (*)
│ │ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ │ ├── aws-smithy-http-tower v0.51.0
│ │ │ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-sdk-sso v0.21.0
│ │ ├── aws-endpoint v0.51.0
│ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ │ ├── aws-types v0.51.0 (*)
│ │ ├── aws-http v0.51.0 (*)
│ │ ├── aws-sig-auth v0.51.0
│ │ │ ├── aws-sigv4 v0.51.0
│ │ │ │ ├── aws-smithy-eventstream v0.51.0 (*)
│ │ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-smithy-eventstream v0.51.0 (*)
│ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-types v0.51.0 (*)
│ │ ├── aws-smithy-async v0.51.0 (*)
│ │ ├── aws-smithy-client v0.51.0 (*)
│ │ ├── aws-smithy-http v0.51.0 (*)
│ │ ├── aws-smithy-http-tower v0.51.0 (*)
│ │ ├── aws-smithy-json v0.51.0
│ │ │ └── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-types v0.51.0 (*)
│ ├── aws-sdk-sts v0.21.0
│ │ ├── aws-endpoint v0.51.0 (*)
│ │ ├── aws-http v0.51.0 (*)
│ │ ├── aws-sig-auth v0.51.0 (*)
│ │ ├── aws-smithy-async v0.51.0 (*)
│ │ ├── aws-smithy-client v0.51.0 (*)
│ │ ├── aws-smithy-http v0.51.0 (*)
│ │ ├── aws-smithy-http-tower v0.51.0 (*)
│ │ ├── aws-smithy-query v0.51.0
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-xml v0.51.0
│ │ ├── aws-types v0.51.0 (*)
│ ├── aws-smithy-async v0.51.0 (*)
│ ├── aws-smithy-client v0.51.0 (*)
│ ├── aws-smithy-http v0.51.0 (*)
│ ├── aws-smithy-http-tower v0.51.0 (*)
│ ├── aws-smithy-json v0.51.0 (*)
│ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-types v0.51.0 (*)
├── aws-sdk-s3 v0.21.0
│ ├── aws-endpoint v0.51.0 (*)
│ ├── aws-http v0.51.0 (*)
│ ├── aws-sig-auth v0.51.0 (*)
│ ├── aws-sigv4 v0.51.0 (*)
│ ├── aws-smithy-async v0.51.0 (*)
│ ├── aws-smithy-checksums v0.51.0
│ │ ├── aws-smithy-http v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-smithy-client v0.51.0 (*)
│ ├── aws-smithy-eventstream v0.51.0 (*)
│ ├── aws-smithy-http v0.51.0 (*)
│ ├── aws-smithy-http-tower v0.51.0 (*)
│ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-smithy-xml v0.51.0 (*)
│ ├── aws-types v0.51.0 (*)
├── aws-smithy-http v0.51.0 (*)
├── aws-types v0.51.0 (*)
```

### Environment details (OS name and version, etc.)

Linux localhost 5.15.0-48-generic #54-Ubuntu SMP Fri Aug 26 13:26:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

### Logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in sdk/s3/src/http_body_checksum.rs at the validation that rejects request bodies without a known size, then trace how PutObject converts the hyper::Body through SdkBody and ByteStream. Confirm how the supplied content_length relates to the body size hint, and add coverage showing that a streaming body with a known length can be checksum validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.