[Improvement] Share Paimon historical lookup disk cache across tables
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
## Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
## Description
Historical lookups against Paimon currently create one `IOManager` for each `PaimonLakeTableLookuper`. The TabletServer-level lookup-cache disk budget is divided evenly among at most ten cached table lookupers, and each lookuper passes its share to Paimon's `lookup.cache-max-disk-size` option.
This table-level allocation has two drawbacks:
- Cold tables retain a fixed share of the disk budget while a hot table cannot use the unused capacity.
- Reclaiming space requires evicting or closing a whole table lookuper even when evicting a few cold data-file cache entries would be sufficient.
Move Paimon historical lookup-cache ownership to a shared manager scoped to one Fluss TabletServer/lake-storage instance. The shared manager should:
1. Enforce one disk budget across all Paimon table lookupers on the instance.
2. Account for the actual on-disk size of materialized lookup files rather than reserving a fixed amount per table.
3. Identify cached entries by table and Paimon data-file identity, and update recency when a data file is used.
4. Evict cold entries at data-file granularity using LRU and/or TTL when admitting a new file would exceed the global budget.
5. Avoid deleting a file while an in-flight lookup is using it, for example through reference counting or deferred deletion.
6. Remove stale entries when a table lookuper is invalidated and clean orphan files during startup.
7. Participate in `LocalDiskManager` write protection so no new lookup files are downloaded while the data disk is write-locked.
8. Continue exposing the shared cache's disk usage and eviction metrics at the TabletServer scope.
After the shared cache is available, remove the per-table lookup-cache size from `LakeStorage.LookuperContext` and stop configuring Paimon's `lookup.cache-max-disk-size` independently for every table lookuper. The in-memory table-lookuper cache may keep its own lifecycle limit, but it should no longer determine ownership or eviction of cached data files.
## Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.