[FIP-28] Support incremental cleanup of historical KV state
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Follow-up to #4119 and #4120. This also follows the design discussion in [PR #4120](https://github.com/apache/fluss/pull/4120#discussion_r3889842050).
Historical partition writes keep local KV state for records that have not yet been incorporated into the lake table. This state currently has no incremental reclamation mechanism and can grow indefinitely for continuously written historical buckets.
An all-or-nothing cleanup was considered in #4120: wait until the lake log end offset exactly matches the local log end offset, then drop and rebuild the whole RocksDB instance. That approach was removed because a continuously written bucket may never observe exact equality, cleanup depends on several asynchronous steps completing together, and rebuilding the entire KV state creates a reclamation cliff and a rebuild-failure window.
Implement incremental cleanup using the RocksDB compaction-filter infrastructure:
- Tag historical values and tombstones with the WAL offset that produced them.
- Maintain a monotonic cleanup watermark for each historical bucket from confirmed lake log end-offset progress. Since the lake log end offset is exclusive and recovery starts from that offset, entries tagged with offsets lower than the watermark are covered by lake storage and are eligible for removal.
- Reuse or generalize the existing TTL compaction-filter support, which already accepts a server-controlled callback, so the cleanup decision is driven by the lake watermark instead of wall-clock time.
- Trigger RocksDB compaction after the watermark advances. Coalesce and rate-limit compaction requests because a compaction filter only physically removes eligible entries while compaction runs.
- Preserve lookup correctness for local values and tombstones while allowing keys already covered by lake to fall back to the refreshed lake lookuper.
- Restore the watermark safely across leader changes and restarts without requiring the lake and local log end offsets to become equal.
Add coverage for continuously written buckets, values and tombstones, repeated lake-progress notifications, leader changes/recovery, compaction rate limiting, and lookup correctness before and after cleanup.
This follow-up does not need to block the 1.0 release.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the existing TTL compaction-filter support and the historical-bucket lake-progress handling. Trace how values and tombstones are written, how watermarks survive leader changes and recovery, and how lookup falls back to lake storage. Done means incremental cleanup works for continuous writes, repeated progress notifications, restarts, and rate-limited compaction without breaking lookups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100