apache / apache/fluss

Clean orphan children under unknown databases

Open
#3,488 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

## Background

Table and partition directories include stable IDs in their remote paths, so the cleanup action can apply an ID-based guard:

- the parsed ID is not in the active ID set
- the parsed ID is no greater than the last-known max ID
- metadata enumeration must be complete

Database directories are different. Their remote paths only contain the database name and do not include a database ID. A database directory that is not in the current metadata snapshot is an **unknown database** — it may be truly orphaned (dropped), or it may be newly created and not yet enumerated.

## Challenge

Database-level orphan cleanup cannot be safely inferred from filesystem paths alone. Without an additional ownership signal, deleting a database directory may accidentally delete data that belongs to a newly recreated database with the same name.

However, even when we cannot determine whether a database directory itself is orphan, we **can** still safely clean orphan table/partition directories underneath it — because table and partition IDs are globally unique and monotonically increasing. The same ID-based guard that protects known databases also applies to unknown ones.

## Approach

For database directories whose ownership status is unknown:

1. **Never delete the database directory itself** — we cannot safely determine if it is orphan.
2. **Apply the standard ID-based orphan detection** to table/partition subdirectories under it — if a table or partition ID satisfies the orphan criteria (not in active set, not beyond max known ID, metadata complete), it is safe to clean.
3. **Respect existing opt-in flags** — `--allow-clean-orphan-tables` and `--allow-clean-orphan-partitions` control whether these children get cleaned.

This gives us incremental cleanup coverage without risking accidental deletion of live data.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Find the orphan-cleanup implementation and trace the existing ID-based guard for table and partition directories. Check how --allow-clean-orphan-tables, --allow-clean-orphan-partitions, and metadata completeness are handled, then use the existing cleanup tests or add focused coverage. Done means unknown database directories remain, while eligible orphan children are cleaned only when their flags permit it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.