apache / apache/arrow-rs-object-store

object_store: Support for streaming body in `put/put_opts`

Open
#281 13 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently, `put/put_opts` takes a `PutPayload` which is just a wrapper around an array of `Bytes`. It would be useful (for us) to be able to stream the body without having to do a multipart upload.

**Describe the solution you'd like**

The underlying http library `reqwest` already supports this. So I think we can change the definition of `PutPayload` to an enum like so:

```
type StreamingBody = Box>> + Send + 'static>;

pub enum PutPayload {
Bytes(Arc<[Bytes]>),
Streaming(StreamingBody,usize),
}
```

where `PutPayload::Streaming` holds the stream and the total content length (which must be known ahead of time).

The (unfortunate) requirement of the S3 API to know the `content-length` up front means this isn't fully general but can be useful in certain situations such as buffering a largeish file on disk before uploading. In that case, a streaming body would allow the upload to not have to load the entire object in memory to upload it.

**Describe alternatives you've considered**

This can already be done with `put_multipart` but mutli-part upload is complicated, may have some performance implications for reads.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start at the object_store put/put_opts entry points and the PutPayload definition, then trace how payloads reach the underlying reqwest request. Compare this with put_multipart and verify the existing upload tests or add coverage for a known-length streaming body; done means streaming uploads work without buffering the entire object and existing byte uploads remain supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.