Clean orphan children under unknown databases
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
## Background
Table and partition directories include stable IDs in their remote paths, so the cleanup action can apply an ID-based guard:
- the parsed ID is not in the active ID set
- the parsed ID is no greater than the last-known max ID
- metadata enumeration must be complete
Database directories are different. Their remote paths only contain the database name and do not include a database ID. A database directory that is not in the current metadata snapshot is an **unknown database** — it may be truly orphaned (dropped), or it may be newly created and not yet enumerated.
## Challenge
Database-level orphan cleanup cannot be safely inferred from filesystem paths alone. Without an additional ownership signal, deleting a database directory may accidentally delete data that belongs to a newly recreated database with the same name.
However, even when we cannot determine whether a database directory itself is orphan, we **can** still safely clean orphan table/partition directories underneath it — because table and partition IDs are globally unique and monotonically increasing. The same ID-based guard that protects known databases also applies to unknown ones.
## Approach
For database directories whose ownership status is unknown:
1. **Never delete the database directory itself** — we cannot safely determine if it is orphan.
2. **Apply the standard ID-based orphan detection** to table/partition subdirectories under it — if a table or partition ID satisfies the orphan criteria (not in active set, not beyond max known ID, metadata complete), it is safe to clean.
3. **Respect existing opt-in flags** — `--allow-clean-orphan-tables` and `--allow-clean-orphan-partitions` control whether these children get cleaned.
This gives us incremental cleanup coverage without risking accidental deletion of live data.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Find the orphan-cleanup implementation and trace the existing ID-based guard for table and partition directories. Check how --allow-clean-orphan-tables, --allow-clean-orphan-partitions, and metadata completeness are handled, then use the existing cleanup tests or add focused coverage. Done means unknown database directories remain, while eligible orphan children are cleaned only when their flags permit it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100