apache / apache/fluss

[FIP-28] Support incremental cleanup of historical KV state

Open
#4,159 1 comment 0 reactions 0 assignees View on GitHub
component=server priority=normal
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Follow-up to #4119 and #4120. This also follows the design discussion in [PR #4120](https://github.com/apache/fluss/pull/4120#discussion_r3889842050).

Historical partition writes keep local KV state for records that have not yet been incorporated into the lake table. This state currently has no incremental reclamation mechanism and can grow indefinitely for continuously written historical buckets.

An all-or-nothing cleanup was considered in #4120: wait until the lake log end offset exactly matches the local log end offset, then drop and rebuild the whole RocksDB instance. That approach was removed because a continuously written bucket may never observe exact equality, cleanup depends on several asynchronous steps completing together, and rebuilding the entire KV state creates a reclamation cliff and a rebuild-failure window.

Implement incremental cleanup using the RocksDB compaction-filter infrastructure:

- Tag historical values and tombstones with the WAL offset that produced them.
- Maintain a monotonic cleanup watermark for each historical bucket from confirmed lake log end-offset progress. Since the lake log end offset is exclusive and recovery starts from that offset, entries tagged with offsets lower than the watermark are covered by lake storage and are eligible for removal.
- Reuse or generalize the existing TTL compaction-filter support, which already accepts a server-controlled callback, so the cleanup decision is driven by the lake watermark instead of wall-clock time.
- Trigger RocksDB compaction after the watermark advances. Coalesce and rate-limit compaction requests because a compaction filter only physically removes eligible entries while compaction runs.
- Preserve lookup correctness for local values and tombstones while allowing keys already covered by lake to fall back to the refreshed lake lookuper.
- Restore the watermark safely across leader changes and restarts without requiring the lake and local log end offsets to become equal.

Add coverage for continuously written buckets, values and tombstones, repeated lake-progress notifications, leader changes/recovery, compaction rate limiting, and lookup correctness before and after cleanup.

This follow-up does not need to block the 1.0 release.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing TTL compaction-filter support and the historical-bucket lake-progress handling. Trace how values and tombstones are written, how watermarks survive leader changes and recovery, and how lookup falls back to lake storage. Done means incremental cleanup works for continuous writes, repeated progress notifications, restarts, and rate-limited compaction without breaking lookups.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.