aws / aws/aws-sdk-ruby

S3 put_object should accept a block to facillitate chunked writes

Open
#3,142 7 comments 0 reactions 0 assignees View on GitHub
feature-request needs-major-version
Dominant language
Ruby
Stars
3.7k
Forks
1.2k
Avg merge
1d 4h
Merged PRs (30d)
5

Description

### Describe the feature

After using `get_object`'s chunked read, I assumed `put_object` similarly supported chunked writing:

```rb
client.put_object(bucket: blob.bucket, key: blob.key) do |buffer|
while chunk = blob.read(16 * 256)
buffer << chunk
end
end
```

For reference, `get_object` supports this:

```rb
client.get_object(bucket: blob.bucket, key: blob.key) do |chunk|
buffer << chunk
end
```

But this isn't currently supported and results in an empty object, since the block is ignored.

### Use Case

I want to write an IO to S3 while maintaining a low memory footprint, while being explicit with how much I read for each chunk. I do not want to rely on S3 internals to choose how large my chunks should be.

### Proposed Solution

Similarly to `get_object`, allow `put_object` to accept a block, yielding the internal request body.

### Other Information

_No response_

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### SDK version used

1.113.0

### Environment details (OS name and version, etc.)

Linux 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.