datafusion-contrib / datafusion-contrib/liquid-cache
Io_uring based IO for evicted liquid arrays
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
Currenlty we issue blocking read/write to disk: https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L163
https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L443
This blocks the entire runtime. We want to explore how io_uring can help performance.
Prior art: https://github.com/tokio-rs/tokio-uring
https://github.com/tokio-rs/tokio/discussions/4703
We want a benchmark where x-axis is thread count, y axis is io-usage.
To measure disk usage: https://docs.rs/sysinfo/latest/sysinfo/struct.DiskUsage.html
Also want to plot the latency.
To vary the thread count: https://github.com/XiangpengHao/liquid-cache/blob/main/benchmark/clickbench/clickbench_client.rs#L166
=====
We want to compare:
- current approach
- tokio::fs::File
- your io_uring implementation. Blocking io_uring; thread pool based io_uring; and non-blocking io_uring.
=====
Run clickbench, hits.parquet ~15GB.
Run query 20: https://github.com/XiangpengHao/liquid-cache/blob/main/benchmark/clickbench/queries/queries.sql#L21
Paper references:
https://vldb.org/pvldb/vol16/p2090-haas.pdf
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the blocking reads and writes in src/liquid_parquet/src/cache/mod.rs and thread-count handling in benchmark/clickbench/clickbench_client.rs. Run ClickBench query 20 against hits.parquet, then compare the current approach, tokio::fs::File, and the requested io_uring variants. Done means benchmarks varying thread count with disk usage and latency plots.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- operating-systems, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100