datafusion-contrib / datafusion-contrib/liquid-cache
How to manage disk cache (t4) growth in local mode?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
We're running LiquidCache in local mode (Kubernetes, max_cache_bytes set to 30% of RAM).
After running for few hours, the liquid_cache.t4 file has grown to 24GB and doesn't appear to shrink.
From reading the code, it looks like LiquidPolicy::find_victim() only evicts from memory queues and
never drains QueueKind::Disk. BudgetAccounting tracks used_disk_bytes but there doesn't seem to be a
max_disk_bytes cap.
A few questions:
- Is there an existing way to limit or manage disk cache size that we're missing?
- If not, would something like a
max_disk_bytesknob onLiquidCacheLocalBuilderbe the right approach,
or is there a different design direction in mind? - With
AlwaysHydrate, re-accessed entries get promoted to memory and then re-evicted back to
disk as new entries. Is this expected to contribute to disk growth over time?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading LiquidPolicy::find_victim(), QueueKind::Disk, and BudgetAccounting to trace how disk usage is accounted for and evicted. Then inspect LiquidCacheLocalBuilder and the AlwaysHydrate behavior described in the issue. Done means establishing whether disk growth is expected and agreeing on a supported limit or management design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100