awslabs / awslabs/aws-sdk-rust
Allow non-retryable sources when uploading files to S3
- Dominant language
- Rust
- Stars
- 3.3k
- Forks
- 290
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 3
Description
### Describe the feature
Ability to build a `ByteStream` from an implementation of `AsyncBufRead`.
### Use Case
A simple use-case is, we want to read some data from an existing stream (prior S3 download, for example), compress it on the fly, and upload it.
Example code that currently does NOT work:
```rust
let get_resp = aws_client
.get_object()
.bucket(AWS_BUCKET_NAME)
.key(AWS_RBL_PATH)
.send()
.await?;
let read = StreamReader::new(get_resp.body.map_err(convert_smithy_error));
let z_encoder = ZstdEncoder::with_quality(read, Level::Precise(5));
let put_resp = s3_client
.put_object()
.bucket(S3_BUCKET)
.key(S3_DESTINATION_FILE)
.body(z_encoder)
.send()
.await?;
```
If we assume we don't want retries, it should be possible to use a simple AsyncBufRead implementation for the PUT operation.
### Proposed Solution
_No response_
### Other Information
_No response_
### Acknowledgements
- [ ] I may be able to implement this feature request
- [X] This feature might incur a breaking change
### A note for the community
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue, please leave a comment
Contributor guide
Research direction
Start by reading the ByteStream and AsyncBufRead entry points, then trace how the put_object body handles retryable and non-retryable sources. Use the provided StreamReader and ZstdEncoder example to verify that a compressed existing stream can be uploaded without retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100