apache / apache/arrow-rs-object-store

object_store: Using `io_uring`?

Open
#262 7 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Which part is this question about**
`object_store`'s code.

**Describe your question**
For [Zarr](https://zarr.dev/), we may want to read on the order of 1 million parts of files per second (from a single machine). It's possible that the only way to achieve this performance will be to use [`io_uring`](https://en.wikipedia.org/wiki/Io_uring) to send many IO operations to the Linux kernel using just a single system call.

Would `object_store` ever consider implementing an async `io_uring` backend for [`get_ranges`](https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#method.get_ranges)? (I may be able to write the PR, with some hand-holding!)

**Additional context**
[io_uring](https://news.ycombinator.com/item?id=23133040) is a newish feature of the Linux kernel that allows for requesting many IO operations with a single system call - including local file operations and network operations - without any memory copying, and with minimal system calls. [Some database folks](https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/) seem pretty excited about io_uring. [Some benchmarks](https://www.phoronix.com/news/Linux-5.6-IO-uring-Tests) show that io_uring can deliver almost 20x more IOPs for random reads than the previous approach.

Contributor guide

Open the contributing guide

Research direction

Start with the object_store code and the ObjectStore::get_ranges entry point linked in the issue, then review how async I/O backends are structured. Investigate whether Linux io_uring can support the proposed workload and define what an acceptable backend would need to provide; the issue is currently a feasibility question rather than a scoped change.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
backend, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.