apache / apache/arrow-rs-object-store

Add ObjectStore::list_opts

Open
#295 4 comments 3 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

We should add an ObjectStore::list_opts to allow

* Specifying both a start offset and delimiter apache/arrow-rs-object-store#291
* Providing `Extensions` apache/arrow-rs#7213

**Describe the solution you'd like**

I _think_ the following API should work and eventually allow providing default implementations of `list` and `list_with_delimiter` that call through to `list_opts`

```
struct ListOptions {
pub offset: Option,
pub delimiter: bool,
pub extensions: Extensions
}

fn list_opts(&self, opts: ListOptions) -> BoxStream<'static, Result>
```

**Describe alternatives you've considered**

We could expose an API that more explicitly exposes the paginated nature of the underlying APIs, however, such APIs are quite cumbersome to use, and wouldn't obviously translate to LocalFileSystem.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the ObjectStore trait and its existing list and list_with_delimiter APIs, then review the context in apache/arrow-rs-object-store#291 and apache/arrow-rs#7213. Done means the proposed ListOptions and list_opts API are integrated with the requested offset, delimiter, and Extensions inputs, with the existing listing behavior accounted for.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.