lance-format / lance-format/lance
Add time-to-idle (TTL) to LanceCache and revisit eviction policy for retired/old-version entries
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Summary
Add an optional time-to-idle (TTL) setting to LanceCache so that cache entries
which are no longer referenced — most notably index entries for a UUID that was
retired by optimize_indices(...retrain...) — are reclaimed after a period of
inactivity, instead of lingering until capacity-based eviction.
This is a memory-retention improvement, not a correctness fix. Index cache
entries are keyed by index UUID (InternalCacheKey), so a retrained index gets a
new UUID and therefore a new key. A read of the current version never collides
with the retired UUID's entry — the stale entries are simply dead weight against
the cache's capacity budget.
Supersedes #7094, which proposed immediate exact-key invalidation of retired
index entries on commit. That approach hurts readers that legitimately use the
old version:
- In-flight MVCC queries that started before the write may still reference the
old index. - Readers pinned to an older table version while writers run multiple operations.
A TTL lets those entries stay cached while still in use and expire naturally once
nobody touches them.
Phase 1 — time-to-idle on LanceCache (primary deliverable)
moka 0.12 (moka::future::Cache) supports this natively via
.time_to_idle(Duration), already used elsewhere in the repo
(rust/lance-io/src/uring/reader.rs).
Decisions:
time_to_idle, nottime_to_live. Idle-based expiry resets on access, so
an old-version index stays cached as long as a pinned reader keeps querying it,
and is reclaimed only after it goes quiet. Absolute TTL would evict an
actively-used pinned index mid-use — reintroducing the very regression we want
to avoid.- Global (per-cache) TTL, not per-entry. moka's builder TTL is per-cache;
that is sufficient. (Per-entryExpiryis noted in Phase 2.) - Default: no TTL. Preserves current behavior; opt-in only.
- Composes with the existing weighted-capacity LRU/weigher — no entry-struct or
codec changes needed.
Surface:
- Thread the TTL through
MokaCacheBackend/LanceCacheconstruction
(e.g. awith_capacity_and_ttlor builder variant). - Expose it at the
Sessionlevel (configures both the index and metadata
caches). Note: moka fixestime_to_idleat cache build time — it cannot
be changed on a live cache. So the Session knob must feed cache construction
(constructor arg / builder method), not mutate an existing cache. - Keep parameter names consistent across Rust / Python / Java per repo
guidelines.
Behavior note: moka expiration is lazy — memory is reclaimed when pending tasks
run (run_pending_tasks), not at the exact instant of expiry. Acceptable for a
GC mechanism.
Phase 2 — eviction ordering / new-vs-old priority (design exploration)
Separate, harder problem surfaced while scoping this: under capacity pressure,
moka's default TinyLFU policy can favor evicting a freshly inserted entry
over an older, historically more-frequently-queried one — and may even reject
the new entry on admission until it builds up frequency. For the
old-segment / new-segment case this is suboptimal: the new segment usually gets
the most traffic going forward, but has no accumulated frequency yet (the
outcome depends on TinyLFU's frequency-decay rate).
We'd like a way to express "this entry is likely to be hit in the near future."
moka 0.12.15 levers:
.eviction_policy(EvictionPolicy::lru())— pure recency-LRU. A new entry
lands at MRU and is evicted last, directly addressing the scenario. Cost: loses
TinyLFU's scan-resistance (a large sequential scan can flush the working set).
Global, not per-entry..expire_after(impl Expiry)— per-entry TTL; lets an old-version entry
expire sooner than a current one. But this is a retention knob (time), not an
eviction-order knob (who loses under pressure).- moka exposes no per-entry priority / "boost" API; you cannot seed an
entry's frequency or pin eviction order without switching the whole policy.
Phase 2 is to evaluate these and decide whether a policy choice (TinyLFU vs LRU)
and/or per-entry retention belongs in the Session config, or whether this needs
something moka doesn't offer.
Open questions
- One TTL for both caches, or separate knobs for index vs metadata cache?
- Phase 2: is
EvictionPolicy::lru()an acceptable global tradeoff, or is the
scan-resistance loss a dealbreaker for the metadata cache? - Phase 2: is per-entry retention (
Expiry) worth the complexity, or is a global
policy enough?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the MokaCacheBackend and LanceCache construction paths, the Session cache configuration, and rust/lance-io/src/uring/reader.rs for the existing moka time-to-idle usage. Thread an optional construction-time TTL through both caches while preserving the no-TTL default, then verify lazy expiration and Session behavior. Separately evaluate the Phase 2 eviction-policy options and record the resulting decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, rust
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100