datafusion-contrib / datafusion-contrib/liquid-cache
Cache simulator
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
We want to study the how different evict/promote policies impact performance. In order to do that, we need a cache trace and a cache simulator that can replay the trace.
To collect cache trace, we need to instructment all the cache get operations:
- Get arrow array with filter https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L266
- Evaluate selection with predicate: https://github.com/XiangpengHao/liquid-cache/blob/main/src/liquid_parquet/src/cache/mod.rs#L206
We will get a trace like this:
- Get(RowGroup id, column id, row id, size)
- Get(RowGroup id, column id, row id, size)
- Get(RowGroup id, column id, row id, size)
One we have the trace, we want to build a cache simulator which explores:
- Eviction policy: when and which batch to evict?
- Promotion policy: should we insert the missed batch to cache?
- How does cache size impact performance
Simulator will have one interface: get(RowGroup id, column id, row id) -> Option.
On cache hit, it returns Some(V); on miss it returns None.
Internally, the simulator decides whether to insert the (RowGroup id, column id, row id) to cache.
Reference: https://blog.xiangpeng.systems/posts/parquet-pushdown/
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the cache get operations in src/liquid_parquet/src/cache/mod.rs at the linked locations around lines 206 and 266, and read the referenced cache-trace design. Define how traces record row-group, column, row, and size values, then determine the simulator behavior for get(RowGroup id, column id, row id) across eviction, promotion, and cache-size policies. Done means a trace can be replayed and policy performance can be compared.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100