apache / apache/arrow-rs-object-store
Remove file-handle from object store GET operations
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
# What
Remove [`GetResultPayload::File`](https://docs.rs/object_store/latest/object_store/enum.GetResultPayload.html#variant.File) (and the entire enum).
# Why
The file handle has several drawbacks as it does NOT specify the following properties:
- **mode:** Is the file opened read-only or does it also have write access (which would be bad)?
- **seek position:** At which position is the current file "read head"? Are users expected to seek to the file start if they want to read the entire file?
- **direct IO:** Is the file opened using direct IO? (in which case the user would have to read in aligned blocks)
- **exclusive access:** Does the API user have exclusive access to the said file handle (and hence can use seek-based operations) or not (in which case only non-seeking operations like [`read_at`](https://doc.rust-lang.org/std/os/unix/fs/trait.FileExt.html#tymethod.read_at) would be required)
At the same time I suspect that most users copy the file data into a buffer anyways and don't use any kernel-based IO operations like [`sendfile`](https://man.archlinux.org/man/sendfile.2).
All object store users and integrating libraries potentially have to deal with streams AND file handles.
# How
Remove the entire [`GetResultPayload`](https://docs.rs/object_store/latest/object_store/enum.GetResultPayload.html) enum and just use streams.
Contributor guide
Assessment
This issue has not been assessed yet.