datafusion-contrib / datafusion-contrib/liquid-cache

Io_uring based IO for evicted liquid arrays

Open
#147 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Rust
Stars
452
Forks
51
Avg merge
3h 10m
Merged PRs (30d)
12

Description

Currenlty we issue blocking read/write to disk: https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L163
https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L443

This blocks the entire runtime. We want to explore how io_uring can help performance.

Prior art: https://github.com/tokio-rs/tokio-uring
https://github.com/tokio-rs/tokio/discussions/4703

We want a benchmark where x-axis is thread count, y axis is io-usage.

To measure disk usage: https://docs.rs/sysinfo/latest/sysinfo/struct.DiskUsage.html
Also want to plot the latency.
To vary the thread count: https://github.com/XiangpengHao/liquid-cache/blob/main/benchmark/clickbench/clickbench_client.rs#L166

=====

We want to compare:

  1. current approach
  2. tokio::fs::File
  3. your io_uring implementation. Blocking io_uring; thread pool based io_uring; and non-blocking io_uring.

=====

Run clickbench, hits.parquet ~15GB.
Run query 20: https://github.com/XiangpengHao/liquid-cache/blob/main/benchmark/clickbench/queries/queries.sql#L21

https://github.com/XiangpengHao/liquid-cache/blob/main/benchmark/clickbench/clickbench_client.rs#L134

Paper references:
https://vldb.org/pvldb/vol16/p2090-haas.pdf

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the blocking reads and writes in src/liquid_parquet/src/cache/mod.rs and thread-count handling in benchmark/clickbench/clickbench_client.rs. Run ClickBench query 20 against hits.parquet, then compare the current approach, tokio::fs::File, and the requested io_uring variants. Done means benchmarks varying thread count with disk usage and latency plots.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
operating-systems, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.