apache / apache/fluss

[Improvement] Share Paimon historical lookup disk cache across tables

Open
#3,955 2 comments 0 reactions 1 assignee Claimed by @zuston View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

## Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

## Description

Historical lookups against Paimon currently create one `IOManager` for each `PaimonLakeTableLookuper`. The TabletServer-level lookup-cache disk budget is divided evenly among at most ten cached table lookupers, and each lookuper passes its share to Paimon's `lookup.cache-max-disk-size` option.

This table-level allocation has two drawbacks:

- Cold tables retain a fixed share of the disk budget while a hot table cannot use the unused capacity.
- Reclaiming space requires evicting or closing a whole table lookuper even when evicting a few cold data-file cache entries would be sufficient.

Move Paimon historical lookup-cache ownership to a shared manager scoped to one Fluss TabletServer/lake-storage instance. The shared manager should:

1. Enforce one disk budget across all Paimon table lookupers on the instance.
2. Account for the actual on-disk size of materialized lookup files rather than reserving a fixed amount per table.
3. Identify cached entries by table and Paimon data-file identity, and update recency when a data file is used.
4. Evict cold entries at data-file granularity using LRU and/or TTL when admitting a new file would exceed the global budget.
5. Avoid deleting a file while an in-flight lookup is using it, for example through reference counting or deferred deletion.
6. Remove stale entries when a table lookuper is invalidated and clean orphan files during startup.
7. Participate in `LocalDiskManager` write protection so no new lookup files are downloaded while the data disk is write-locked.
8. Continue exposing the shared cache's disk usage and eviction metrics at the TabletServer scope.

After the shared cache is available, remove the per-table lookup-cache size from `LakeStorage.LookuperContext` and stop configuring Paimon's `lookup.cache-max-disk-size` independently for every table lookuper. The in-memory table-lookuper cache may keep its own lifecycle limit, but it should no longer determine ownership or eviction of cached data files.

## Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.