apache / apache/gravitino

[EPIC] Support durable purge jobs for Iceberg REST deletion V2

Open
#12,258 0 comments 0 reactions 1 assignee Claimed by @nevzheng View on GitHub
epic
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the proposal

This is the purge-execution child epic of #12219. It builds on the relational deletion model in #12220 and the Iceberg REST delete and UNDROP lifecycle in #12221. The implementation follows the API lifecycle stack ending in #12227.

The goal of this epic is to complete Iceberg REST deletion V2: retained tables can be handed off to durable, asynchronous purge jobs after their retention deadline, without deleting Iceberg files on the `DELETE` request thread.

#### End-to-end behavior

1. `DELETE` durably marks a table as deleted and records its retention deadline. While retained, the table remains hidden, its name remains reserved, and it may be restored through UNDROP.
2. A periodic collector finds a bounded number of eligible `DELETED` actions whose retention has expired.
3. In one short relational transaction, the collector revalidates the exact table and deletion generation, creates its linked purge job, and moves the action from `DELETED` to `PURGING`. This transition closes the UNDROP window.
4. Existing asynchronous cleanup workers claim the job, renew its heartbeat lease, and delete the Iceberg registration and referenced files outside the request path.
5. After cleanup succeeds, one transaction completes the linked job and removes only the exact retained metadata generation. The table name may then be reused, and ordinary lookup returns `404` because the deletion has been fully consumed.

#### Consistency and recovery

- `table_meta.deletion_id` and `entity_deletion` remain the lifecycle source of truth. Purge-job state records execution and progress; it does not replace the deletion lifecycle.
- The deletion action and linked job are handed off atomically, so there is no durable `PURGING` action without its job.
- Every linked job carries the immutable table ID and exact deletion ID. Name snapshots are used for routing, never as the sole deletion predicate.
- UNDROP and purge claim the same retained row under a fine-grained relational lock. Exactly one can move the action out of `DELETED`.
- Existing claim, compare-and-set, heartbeat, lease-reclaim, and durable retry behavior make jobs restart-safe across worker and server failures.
- External cleanup is repeatable: missing files are treated as already complete, children are deleted before parents, and the root Iceberg metadata file is deleted last.
- Finalization verifies the linked table ID, deletion ID, lifecycle state, and purge job ID before deleting relational metadata. An older deletion can never purge a later same-name table.
- Manifest progress is advisory operator information, not a checkpoint. It may restart from zero after a lease reclaim without changing correctness.

#### Reuse and compatibility

This work extends the existing `iceberg_cleanup_job`, `IcebergCleanupJobStore`, and `IcebergCleanupManager` table-as-queue implementation. The existing scheduled executor supplies periodic wake-ups; correctness comes from the relational claim and heartbeat lease rather than a new scheduler, generic job framework, or saga.

Legacy V1 cleanup jobs continue through their current path. When deletion V2 is enabled, linked V2 jobs share the same durable queue and workers. Disabling new V2 collection does not abandon work already in `PURGING`; those jobs continue to drain safely.

#### Backward-compatibility contract

- Database changes are additive and must be deployed before deletion V2 is enabled.
- An unlinked cleanup job is a V1 job and retains its existing execution and completion behavior. A V2 purge job is explicitly linked by both table ID and deletion ID; partial linkage is rejected.
- V2-capable workers may process V1 and V2 jobs from the same queue. Enabling V2 does not migrate, reinterpret, or rewrite existing V1 jobs.
- Disabling V2 stops the creation of new linked purge jobs. Existing `PURGING` jobs continue to completion, while retained `DELETED` tables remain hidden and name-reserved rather than falling back to the V1 cleanup path.
- Rollout must upgrade all cleanup workers before V2 begins creating linked jobs. Once linked jobs or retained deletion actions exist, running or rolling back to a V1-only worker is unsupported because it cannot enforce the V2 generation predicates.

#### After this epic

Iceberg REST deletion V2 will support durable purge jobs end to end: bounded expiry collection, atomic ownership transfer, asynchronous cleanup, visible progress, crash-safe retry, exact generation finalization, and compatibility with existing V1 cleanup.

#### Out of scope

- A generic public Jobs API or operator control plane.
- Purge handlers for entity types other than Iceberg REST tables.
- A new scheduler framework or external queue service.
- Long-term audit storage or terminal-record retention policy.
- Full deployment-scale and multi-node performance validation, which belongs to the integration and scale-testing epic.

### Review order

1. #12259 — propagate asynchronous cleanup failures
2. #12260 — report advisory manifest cleanup progress
3. #12261 — atomically hand an expired deletion to its purge job
4. #12262 — remove the retained Iceberg registration before cleanup
5. #12266 — atomically finalize the exact retained generation
6. #12264 — snapshot immutable Iceberg cleanup context
7. #12263 — coordinate bounded retained-deletion cleanup
8. #12265 — wire retained cleanup into the runtime

### Task list

- [ ] #12259 — propagate asynchronous cleanup failures
- [ ] #12260 — report advisory manifest cleanup progress
- [ ] #12261 — atomically hand an expired deletion to its purge job
- [ ] #12262 — remove the retained Iceberg registration before cleanup
- [ ] #12266 — atomically finalize the exact retained generation
- [ ] #12264 — snapshot immutable Iceberg cleanup context
- [ ] #12263 — coordinate bounded retained-deletion cleanup
- [ ] #12265 — wire retained cleanup into the runtime

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.