datafusion-contrib / datafusion-contrib/liquid-cache
Support cache pass-through for selective scans
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
It would be useful to have a way to selectively pass through certain scans without caching.
When working with a mix of small and large datasets, caching everything can cause the t4 file to grow significantly (decoded Arrow batches are ~5x larger than compressed parquet). This leads to memory pressure and degrades cache
lookup performance for smaller datasets that benefit most from caching.
Having a way to control this — for example, a size threshold, a predicate callback, or a configuration option — would allow large datasets (GBs) to pass through without entering the cache, while smaller frequently-accessed data continues to be cached effectively in mixed workloads.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the scan and cache insertion paths, then determine how a size threshold, predicate callback, or configuration option should control pass-through behavior. Done means large selected scans bypass caching while smaller datasets retain effective caching without excessive cache growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100