apache / apache/gravitino

[EPIC] Support durable soft deletion and recovery

Open
#12,219 0 comments 0 reactions 1 assignee Claimed by @nevzheng View on GitHub
epic
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### Describe the proposal

Add durable soft deletion and recovery to Gravitino, beginning with Iceberg REST tables.

A soft-deleted table is hidden from normal use while its original metadata row is retained for a configured recovery period. During that period, an authorized user can restore the same table identity. Physical cleanup happens asynchronously after the recovery period in a later delivery phase.

## Product invariants

- The original metadata row remains the source of identity, name, parent, ownership, and related metadata while retained.
- Each active deletion generation has an opaque `deletion_id` referenced by the source row.
- The deleted name remains reserved until the table is restored or finally purged.
- Restoration reactivates the original object; it does not create a replacement.
- DELETE never removes Iceberg files on the request thread.
- Existing behavior remains unchanged when soft deletion is disabled.
- Additional entity tables may later adopt the same opaque deletion-pointer pattern. Entity-specific purge context and handler dispatch remain pending design.
- Terminal history belongs in emitted audit/change events, not permanent `entity_deletion` receipts or a new deletion-specific audit table.

## Delivery plan

| Area | Tracking | Milestone |
|---|---|---|
| Relational storage | #12220 | Represent, retain, and reactivate the original table row transactionally. |
| Iceberg REST API | #12221 | DELETE, deleted-name discovery, name reservation, and name-only UNDROP. |
| Durable purge jobs | #12258 | Claim expired deletions and remove Iceberg resources asynchronously with progress and retry. |
| Integration and scale validation | Pending | Exercise concurrency, restart, large-table cleanup, and the required scale scenarios. |
| Additional entity types | Pending | Apply the deletion-pointer pattern beyond Iceberg REST tables. |

## First milestone

After #12220 and #12221 land, the retained-metadata user journey works end to end: DELETE retains and hides the original table, `deleted=true` discovers its name, and UNDROP reactivates that same row. Physical purge, durable job progress, terminal audit events, and scale validation remain separate work.

Related work:

- #12187 tracks broader management-API semantics for metadata undelete.
- #11265 introduced asynchronous hard deletion for the Iceberg REST catalog.

### Task list

- [ ] #12220 — relational deletion-generation storage
- [ ] #12221 — Iceberg REST soft-delete and recovery APIs
- [ ] #12258 — durable asynchronous purge execution
- [ ] Cleanup progress, observability, and operator controls — pending design
- [ ] Integration, concurrency, restart, and scale validation — pending design
- [ ] Additional entity types — pending design

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.