apache / apache/arrow-rs-object-store

Object_store: put_file and get_file methods

Open
#267 11 comments 2 reactions 1 assignee Claimed by @midnattsol View on GitHub
enhancement good first issue help wanted
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
As a user, I would like to utilize the object_store to put and retrieve files. Currently, I can employ the put and get methods for this purpose. However, I am responsible for handling any issues that arise. For instance, when uploading a file to the store, I must verify its size to determine whether multipart upload is necessary. Similarly, when retrieving a large file, I cannot download it in its entirety before writing it to the local file system, as this would cause excessive memory usage. Instead, I may need to use get_range and write specific portions of the file locally. Moreover, downloading different ranges could potentially be executed in parallel.

**Describe the solution you'd like**
Expose get_file(local_path, remote_path) and put_file(local_path, remote_path) to be more user-friendly.

**Describe alternatives you've considered**
Stay on what we have now, which means users should implement the functionalities by themself.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.