apache / apache/arrow-rs-object-store

Remove file-handle from object store GET operations

Open
#18 5 comments 0 reactions 1 assignee Claimed by @crepererum View on GitHub
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

# What
Remove [`GetResultPayload::File`](https://docs.rs/object_store/latest/object_store/enum.GetResultPayload.html#variant.File) (and the entire enum).

# Why
The file handle has several drawbacks as it does NOT specify the following properties:

- **mode:** Is the file opened read-only or does it also have write access (which would be bad)?
- **seek position:** At which position is the current file "read head"? Are users expected to seek to the file start if they want to read the entire file?
- **direct IO:** Is the file opened using direct IO? (in which case the user would have to read in aligned blocks)
- **exclusive access:** Does the API user have exclusive access to the said file handle (and hence can use seek-based operations) or not (in which case only non-seeking operations like [`read_at`](https://doc.rust-lang.org/std/os/unix/fs/trait.FileExt.html#tymethod.read_at) would be required)

At the same time I suspect that most users copy the file data into a buffer anyways and don't use any kernel-based IO operations like [`sendfile`](https://man.archlinux.org/man/sendfile.2).

All object store users and integrating libraries potentially have to deal with streams AND file handles.

# How
Remove the entire [`GetResultPayload`](https://docs.rs/object_store/latest/object_store/enum.GetResultPayload.html) enum and just use streams.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.