Support safe cleanup of orphaned KV shared SST files
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Shared SST files are not owned by a single snapshot directory. They may be reused by multiple retained snapshots and still-in-use snapshots. A client-side cleanup action cannot safely decide that a shared SST is orphaned by only checking currently active snapshot IDs or currently visible shared file handles.
The main correctness challenge is concurrency with snapshot lifecycle:
- a snapshot may have uploaded shared SST files but not committed yet
- a snapshot commit result may be temporarily uncertain
- active refs fetched by the cleanup action are only a point-in-time view
- object-store listing and deletion are not protected by the snapshot lifecycle lock
Deleting a shared SST from the cleanup action can therefore race with an in-flight snapshot and cause a later committed snapshot to reference a deleted SST file.
Shared SST cleanup should be implemented with server-side snapshot lifecycle coordination, or with an equivalent protocol that can protect in-flight snapshots.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by tracing the snapshot upload, commit, and cleanup lifecycle; done means shared SST deletion is coordinated with in-flight and uncertain snapshots so a later committed snapshot cannot reference a deleted file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100