datafusion-contrib / datafusion-contrib/liquid-cache

Cache simulator

Open
#145 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Rust
Stars
452
Forks
51
Avg merge
3h 10m
Merged PRs (30d)
12

Description

We want to study the how different evict/promote policies impact performance. In order to do that, we need a cache trace and a cache simulator that can replay the trace.

To collect cache trace, we need to instructment all the cache get operations:

  1. Get arrow array with filter https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L266
  2. Evaluate selection with predicate: https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L206

We will get a trace like this:

  1. Get(RowGroup id, column id, row id, size)
  2. Get(RowGroup id, column id, row id, size)
  3. Get(RowGroup id, column id, row id, size)

One we have the trace, we want to build a cache simulator which explores:

  1. Eviction policy: when and which batch to evict?
  2. Promotion policy: should we insert the missed batch to cache?
  3. How does cache size impact performance

Simulator will have one interface: get(RowGroup id, column id, row id) -> Option.
On cache hit, it returns Some(V); on miss it returns None.

Internally, the simulator decides whether to insert the (RowGroup id, column id, row id) to cache.

Reference: https://blog.xiangpeng.systems/posts/parquet-pushdown/

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cache get operations in src/liquid_parquet/src/cache/mod.rs at the linked locations around lines 206 and 266, and read the referenced cache-trace design. Define how traces record row-group, column, row, and size values, then determine the simulator behavior for get(RowGroup id, column id, row id) across eviction, promotion, and cache-size policies. Done means a trace can be replayed and policy performance can be compared.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.