apache / apache/arrow-rs-object-store
object_store: Support for streaming body in `put/put_opts`
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently, `put/put_opts` takes a `PutPayload` which is just a wrapper around an array of `Bytes`. It would be useful (for us) to be able to stream the body without having to do a multipart upload.
**Describe the solution you'd like**
The underlying http library `reqwest` already supports this. So I think we can change the definition of `PutPayload` to an enum like so:
```
type StreamingBody = Box>> + Send + 'static>;
pub enum PutPayload {
Bytes(Arc<[Bytes]>),
Streaming(StreamingBody,usize),
}
```
where `PutPayload::Streaming` holds the stream and the total content length (which must be known ahead of time).
The (unfortunate) requirement of the S3 API to know the `content-length` up front means this isn't fully general but can be useful in certain situations such as buffering a largeish file on disk before uploading. In that case, a streaming body would allow the upload to not have to load the entire object in memory to upload it.
**Describe alternatives you've considered**
This can already be done with `put_multipart` but mutli-part upload is complicated, may have some performance implications for reads.
**Additional context**
Contributor guide
Research direction
Start at the object_store put/put_opts entry points and the PutPayload definition, then trace how payloads reach the underlying reqwest request. Compare this with put_multipart and verify the existing upload tests or add coverage for a known-length streaming body; done means streaming uploads work without buffering the entire object and existing byte uploads remain supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100