lance-format / lance-format/lance

Add time-to-idle (TTL) to LanceCache and revisit eviction policy for retired/old-version entries

Open
#7,490 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Summary

Add an optional time-to-idle (TTL) setting to LanceCache so that cache entries
which are no longer referenced — most notably index entries for a UUID that was
retired by optimize_indices(...retrain...) — are reclaimed after a period of
inactivity, instead of lingering until capacity-based eviction.

This is a memory-retention improvement, not a correctness fix. Index cache
entries are keyed by index UUID (InternalCacheKey), so a retrained index gets a
new UUID and therefore a new key. A read of the current version never collides
with the retired UUID's entry — the stale entries are simply dead weight against
the cache's capacity budget.

Supersedes #7094, which proposed immediate exact-key invalidation of retired
index entries on commit. That approach hurts readers that legitimately use the
old version:

  1. In-flight MVCC queries that started before the write may still reference the
    old index.
  2. Readers pinned to an older table version while writers run multiple operations.

A TTL lets those entries stay cached while still in use and expire naturally once
nobody touches them.

Phase 1 — time-to-idle on LanceCache (primary deliverable)

moka 0.12 (moka::future::Cache) supports this natively via
.time_to_idle(Duration), already used elsewhere in the repo
(rust/lance-io/src/uring/reader.rs).

Decisions:

  • time_to_idle, not time_to_live. Idle-based expiry resets on access, so
    an old-version index stays cached as long as a pinned reader keeps querying it,
    and is reclaimed only after it goes quiet. Absolute TTL would evict an
    actively-used pinned index mid-use — reintroducing the very regression we want
    to avoid.
  • Global (per-cache) TTL, not per-entry. moka's builder TTL is per-cache;
    that is sufficient. (Per-entry Expiry is noted in Phase 2.)
  • Default: no TTL. Preserves current behavior; opt-in only.
  • Composes with the existing weighted-capacity LRU/weigher — no entry-struct or
    codec changes needed.

Surface:

  • Thread the TTL through MokaCacheBackend / LanceCache construction
    (e.g. a with_capacity_and_ttl or builder variant).
  • Expose it at the Session level (configures both the index and metadata
    caches). Note: moka fixes time_to_idle at cache build time — it cannot
    be changed on a live cache. So the Session knob must feed cache construction
    (constructor arg / builder method), not mutate an existing cache.
  • Keep parameter names consistent across Rust / Python / Java per repo
    guidelines.

Behavior note: moka expiration is lazy — memory is reclaimed when pending tasks
run (run_pending_tasks), not at the exact instant of expiry. Acceptable for a
GC mechanism.

Phase 2 — eviction ordering / new-vs-old priority (design exploration)

Separate, harder problem surfaced while scoping this: under capacity pressure,
moka's default TinyLFU policy can favor evicting a freshly inserted entry
over an older, historically more-frequently-queried one — and may even reject
the new entry on admission until it builds up frequency. For the
old-segment / new-segment case this is suboptimal: the new segment usually gets
the most traffic going forward, but has no accumulated frequency yet (the
outcome depends on TinyLFU's frequency-decay rate).

We'd like a way to express "this entry is likely to be hit in the near future."
moka 0.12.15 levers:

  • .eviction_policy(EvictionPolicy::lru()) — pure recency-LRU. A new entry
    lands at MRU and is evicted last, directly addressing the scenario. Cost: loses
    TinyLFU's scan-resistance (a large sequential scan can flush the working set).
    Global, not per-entry.
  • .expire_after(impl Expiry) — per-entry TTL; lets an old-version entry
    expire sooner than a current one. But this is a retention knob (time), not an
    eviction-order knob (who loses under pressure).
  • moka exposes no per-entry priority / "boost" API; you cannot seed an
    entry's frequency or pin eviction order without switching the whole policy.

Phase 2 is to evaluate these and decide whether a policy choice (TinyLFU vs LRU)
and/or per-entry retention belongs in the Session config, or whether this needs
something moka doesn't offer.

Open questions

  • One TTL for both caches, or separate knobs for index vs metadata cache?
  • Phase 2: is EvictionPolicy::lru() an acceptable global tradeoff, or is the
    scan-resistance loss a dealbreaker for the metadata cache?
  • Phase 2: is per-entry retention (Expiry) worth the complexity, or is a global
    policy enough?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the MokaCacheBackend and LanceCache construction paths, the Session cache configuration, and rust/lance-io/src/uring/reader.rs for the existing moka time-to-idle usage. Thread an optional construction-time TTL through both caches while preserving the no-TTL default, then verify lazy expiration and Session behavior. Separately evaluate the Phase 2 eviction-policy options and record the resulting decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, rust
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.