apache / apache/arrow-rs-object-store
Ability to chunk download from object store
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When downloading large objects (> 300MBs) using object_store crate, I often hit timeout using the default configuration (30 seconds connection timeout). Interestingly, when increasing the timeout, the download speed is actual lower (not sure if it's the same for everyone?).
**Describe the solution you'd like**
I am thinking if it makes sense to chunk a file into smaller ranges (say, 100MB each), and in parallel, download each range with different connection and reconcile them under the same interface.
**Describe alternatives you've considered**
Not sure if such a capability can be composed using the existing interfaces.
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.