datafusion-contrib / datafusion-contrib/liquid-cache

Investigate ways to using io uring in read path

Open
#385 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
452
Forks
51
Avg merge
3h 10m
Merged PRs (30d)
12

Description

We currently only uses io uring in batch eviction: https://github.com/XiangpengHao/liquid-cache/blob/7c6603538321e0d6c0d22ecf76ff1f0b9a9f2229/src/storage/src/cache/core.rs#L423-L432

But the read path is still using the pread/write: https://github.com/XiangpengHao/liquid-cache/blob/7c6603538321e0d6c0d22ecf76ff1f0b9a9f2229/src/parquet/src/cache/io.rs#L146-L171

Concrete items to work on:

  • understand the codebase, and write down a section (3 paragraph) to summarize how we perform io in both read and batch eviction path.
  • investigate ways to use io uring for the read path. Start with easiest and simplest implementation (i.e., not the most performant)

I think @proteetpaul plan to work on this

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading src/storage/src/cache/core.rs around lines 423-432 and src/parquet/src/cache/io.rs around lines 146-171 to compare batch eviction and read-path I/O. Document the current read and eviction flows in three paragraphs, then investigate the simplest way to use io_uring for reads. Done means the summary is written and an initial implementation approach is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.