apache / apache/fluss

Support safe cleanup of orphaned KV shared SST files

Open
#3,489 1 comment 0 reactions 0 assignees View on GitHub
priority=blocker
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

Shared SST files are not owned by a single snapshot directory. They may be reused by multiple retained snapshots and still-in-use snapshots. A client-side cleanup action cannot safely decide that a shared SST is orphaned by only checking currently active snapshot IDs or currently visible shared file handles.

The main correctness challenge is concurrency with snapshot lifecycle:

- a snapshot may have uploaded shared SST files but not committed yet
- a snapshot commit result may be temporarily uncertain
- active refs fetched by the cleanup action are only a point-in-time view
- object-store listing and deletion are not protected by the snapshot lifecycle lock

Deleting a shared SST from the cleanup action can therefore race with an in-flight snapshot and cause a later committed snapshot to reference a deleted SST file.

Shared SST cleanup should be implemented with server-side snapshot lifecycle coordination, or with an equivalent protocol that can protect in-flight snapshots.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by tracing the snapshot upload, commit, and cleanup lifecycle; done means shared SST deletion is coordinated with in-flight and uncertain snapshots so a later committed snapshot cannot reference a deleted file.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.