[EPIC] Support durable soft deletion and recovery
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### Describe the proposal
Add durable soft deletion and recovery to Gravitino, beginning with Iceberg REST tables.
A soft-deleted table is hidden from normal use while its original metadata row is retained for a configured recovery period. During that period, an authorized user can restore the same table identity. Physical cleanup happens asynchronously after the recovery period in a later delivery phase.
## Product invariants
- The original metadata row remains the source of identity, name, parent, ownership, and related metadata while retained.
- Each active deletion generation has an opaque `deletion_id` referenced by the source row.
- The deleted name remains reserved until the table is restored or finally purged.
- Restoration reactivates the original object; it does not create a replacement.
- DELETE never removes Iceberg files on the request thread.
- Existing behavior remains unchanged when soft deletion is disabled.
- Additional entity tables may later adopt the same opaque deletion-pointer pattern. Entity-specific purge context and handler dispatch remain pending design.
- Terminal history belongs in emitted audit/change events, not permanent `entity_deletion` receipts or a new deletion-specific audit table.
## Delivery plan
| Area | Tracking | Milestone |
|---|---|---|
| Relational storage | #12220 | Represent, retain, and reactivate the original table row transactionally. |
| Iceberg REST API | #12221 | DELETE, deleted-name discovery, name reservation, and name-only UNDROP. |
| Durable purge jobs | #12258 | Claim expired deletions and remove Iceberg resources asynchronously with progress and retry. |
| Integration and scale validation | Pending | Exercise concurrency, restart, large-table cleanup, and the required scale scenarios. |
| Additional entity types | Pending | Apply the deletion-pointer pattern beyond Iceberg REST tables. |
## First milestone
After #12220 and #12221 land, the retained-metadata user journey works end to end: DELETE retains and hides the original table, `deleted=true` discovers its name, and UNDROP reactivates that same row. Physical purge, durable job progress, terminal audit events, and scale validation remain separate work.
Related work:
- #12187 tracks broader management-API semantics for metadata undelete.
- #11265 introduced asynchronous hard deletion for the Iceberg REST catalog.
### Task list
- [ ] #12220 — relational deletion-generation storage
- [ ] #12221 — Iceberg REST soft-delete and recovery APIs
- [ ] #12258 — durable asynchronous purge execution
- [ ] Cleanup progress, observability, and operator controls — pending design
- [ ] Integration, concurrency, restart, and scale validation — pending design
- [ ] Additional entity types — pending design
Contributor guide
Assessment
This issue has not been assessed yet.