apache / apache/arrow-rs-object-store
object_store: Using `io_uring`?
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Which part is this question about**
`object_store`'s code.
**Describe your question**
For [Zarr](https://zarr.dev/), we may want to read on the order of 1 million parts of files per second (from a single machine). It's possible that the only way to achieve this performance will be to use [`io_uring`](https://en.wikipedia.org/wiki/Io_uring) to send many IO operations to the Linux kernel using just a single system call.
Would `object_store` ever consider implementing an async `io_uring` backend for [`get_ranges`](https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#method.get_ranges)? (I may be able to write the PR, with some hand-holding!)
**Additional context**
[io_uring](https://news.ycombinator.com/item?id=23133040) is a newish feature of the Linux kernel that allows for requesting many IO operations with a single system call - including local file operations and network operations - without any memory copying, and with minimal system calls. [Some database folks](https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/) seem pretty excited about io_uring. [Some benchmarks](https://www.phoronix.com/news/Linux-5.6-IO-uring-Tests) show that io_uring can deliver almost 20x more IOPs for random reads than the previous approach.
Contributor guide
Research direction
Start with the object_store code and the ObjectStore::get_ranges entry point linked in the issue, then review how async I/O backends are structured. Investigate whether Linux io_uring can support the proposed workload and define what an acceptable backend would need to provide; the issue is currently a feasibility question rather than a scoped change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- backend, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100