awslabs / awslabs/aws-sdk-rust

Allow non-retryable sources when uploading files to S3

Open
#541 3 comments 3 reactions 0 assignees View on GitHub
feature-request p2
Dominant language
Rust
Stars
3.3k
Forks
290
Avg merge
1d 12h
Merged PRs (30d)
3

Description

### Describe the feature

Ability to build a `ByteStream` from an implementation of `AsyncBufRead`.

### Use Case

A simple use-case is, we want to read some data from an existing stream (prior S3 download, for example), compress it on the fly, and upload it.

Example code that currently does NOT work:
```rust
let get_resp = aws_client
.get_object()
.bucket(AWS_BUCKET_NAME)
.key(AWS_RBL_PATH)
.send()
.await?;

let read = StreamReader::new(get_resp.body.map_err(convert_smithy_error));
let z_encoder = ZstdEncoder::with_quality(read, Level::Precise(5));

let put_resp = s3_client
.put_object()
.bucket(S3_BUCKET)
.key(S3_DESTINATION_FILE)
.body(z_encoder)
.send()
.await?;
```

If we assume we don't want retries, it should be possible to use a simple AsyncBufRead implementation for the PUT operation.

### Proposed Solution

_No response_

### Other Information

_No response_

### Acknowledgements

- [ ] I may be able to implement this feature request
- [X] This feature might incur a breaking change

### A note for the community

### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue, please leave a comment

Contributor guide

Open the contributing guide

Research direction

Start by reading the ByteStream and AsyncBufRead entry points, then trace how the put_object body handles retryable and non-retryable sources. Use the provided StreamReader and ZstdEncoder example to verify that a compressed existing stream can be uploaded without retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.