datafusion-contrib / datafusion-contrib/liquid-cache

How to manage disk cache (t4) growth in local mode?

Open
#487 6 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
452
Forks
51
Avg merge
3h 10m
Merged PRs (30d)
12

Description

We're running LiquidCache in local mode (Kubernetes, max_cache_bytes set to 30% of RAM).
After running for few hours, the liquid_cache.t4 file has grown to 24GB and doesn't appear to shrink.

From reading the code, it looks like LiquidPolicy::find_victim() only evicts from memory queues and
never drains QueueKind::Disk. BudgetAccounting tracks used_disk_bytes but there doesn't seem to be a
max_disk_bytes cap.

A few questions:

  1. Is there an existing way to limit or manage disk cache size that we're missing?
  2. If not, would something like a max_disk_bytes knob on LiquidCacheLocalBuilder be the right approach,
    or is there a different design direction in mind?
  3. With AlwaysHydrate, re-accessed entries get promoted to memory and then re-evicted back to
    disk as new entries. Is this expected to contribute to disk growth over time?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading LiquidPolicy::find_victim(), QueueKind::Disk, and BudgetAccounting to trace how disk usage is accounted for and evicted. Then inspect LiquidCacheLocalBuilder and the AlwaysHydrate behavior described in the issue. Done means establishing whether disk growth is expected and agreeing on a supported limit or management design.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.