[PROPOSAL] Opt-in Soft-Delete, Hold Period, and Undrop for Iceberg Tables

add an **opt-in, catalog-scoped soft-delete** for Iceberg tables managed by Polaris:

1. **DROP** (without force-purge) marks the table soft-deleted for a configurable **hold period**.
2. Soft-deleted tables are **not listable or loadable** via normal Iceberg REST paths; the **active name is freed**.
3. Catalog metadata (including metadata location) is **retained** so the table remains recoverable.
4. After the hold (or via privileged force-purge), Polaris performs **permanent delete**, reusing existing drop-with-purge / `ENTITY_CLEANUP_SCHEDULER` machinery for optional data-file cleanup.
5. A follow-up phase adds **list soft-deleted** and **undrop** APIs.

**Default remains today’s hard-drop behavior** (`soft-delete.enabled = false`). No engine changes required for the soft-drop path.

filing this because many organizations use Polaris as the Iceberg REST catalog of record. After moving off catalogs with richer lifecycle tooling, two operational gaps appear:

1. **Compliance / legal hold:** Dropped tables must remain **recoverable for a defined period** before catalog state and data are permanently destroyed.
2. **Polaris today:** Successful table drop removes the **active** catalog entry. There is no first-class undrop or hold period.
- Drop **without** purge: name gone; files may remain on storage but the table is **not** loadable by name.
- Drop **with** purge: schedules async file cleanup when enabled (`DROP_WITH_PURGE_ENABLED`, async `ENTITY_CLEANUP_SCHEDULER`).

Neither path is a recycle bin. Operators invent workarounds (rename into a quarantine namespace; external ledger + `registerTable`). Those work only if every client is forced through a control plane; they fail under direct Iceberg REST access, collide on names, and interact poorly with orphan-file removal.

I feel Polaris is missing is a **productized, opt-in lifecycle** that uses its foundations with a clear API and expiration task :)

### Goals

| ID | Goal |
|----|------|
| G1 | Recoverable drop for a configurable hold period |
| G2 | Safe permanent delete after hold or privileged force-purge |
| G3 | Opt-in; zero behavior change when disabled |
| G4 | Compatible with Iceberg REST clients for soft-drop |
| G5 | Implementable and mergeable in small, reviewable phases |
| G6 | Clear boundary with table maintenance / TMS |

### Non-goals

| ID | Non-goal | Rationale |
|----|----------|-----------|
| NG1 | Catalog-level branching / multi-table git semantics | Separate proposal space ([#3469](https://github.com/apache/polaris/issues/3469)) |
| NG2 | Full table-maintenance execution service (compaction, expiry jobs) | Covered by [#538](https://github.com/apache/polaris/issues/538) / TMS; this RFC only defines lifecycle **state** |
| NG3 | Soft-delete for namespaces, catalogs, principals | Later / different authz story |
| NG4 | Soft-delete for federated / static-facade external catalogs | Remote catalog is source of truth |
| NG5 | Changing Iceberg format or engine SQL grammar | Catalog interprets existing DROP |
| NG6 | Storage Object Lock / immutability | Deployment/storage concern; complementary |
| NG7 | Reintroducing unfinished `entitiesDropped` SPI as-is | Prefer a clean design (as stated in #1070) |

## Prior art and related work

### Historical undrop intent (Polaris)

| Reference | Relevance |
|-----------|-----------|
| [#1070](https://github.com/apache/polaris/pull/1070) | Persistence refactor **removed vestigial `entitiesDropped` methods**: *“originally for UNDROP but ended up not being used; they can be reintroduced with better design in the future if we want to add UNDROP.”* |
| [#766](https://github.com/apache/polaris/issues/766) (comment) | Notes **vestigial plans for UNDROP** when discussing entity index tables; also cites DropTable + TaskEntity for purge cleanup |

This proposal is that: opt-in soft-delete + hold + delayed permanent delete + undrop, not a silent revival of dead SPI methods.

### Related but distinct issues

| Reference | Relationship |
|-----------|----------------|
| [#538](https://github.com/apache/polaris/issues/538) Table maintenance | Soft-deleted tables must be **excluded** from orphan-as-live treatment; TMS remains separate |
| [#774](https://github.com/apache/polaris/issues/774) Task handling incomplete | Permanent purge should use improved task reliability, not invent a second GC |
| [#3008](https://github.com/apache/polaris/issues/3008) Cascading async catalog deletion | Align permanent-delete async patterns where useful |
| Drop+purge bugs ([#289](https://github.com/apache/polaris/issues/289), [#3695](https://github.com/apache/polaris/issues/3695)) | Soft-delete must not schedule file cleanup until permanent delete |
| Roadmap [Discussion #1028](https://github.com/apache/polaris/discussions/1028) | Fits **Data Security, Governance and Compliance**; soft-delete is not currently listed |
| [#3469](https://github.com/apache/polaris/issues/3469) Catalog-level branching | Out of scope |

### Code anchors (current `main`)

- Drop paths: `LocalIcebergCatalog.dropTable`, `IcebergCatalogHandler.dropTableWithoutPurge` / `dropTableWithPurge`
- MetaStore: `dropEntityIfExists` (hard remove from active path; optional cleanup task)
- Cleanup: `AsyncTaskType.ENTITY_CLEANUP_SCHEDULER`, `TableCleanupTaskHandler`
- Config: `DROP_WITH_PURGE_ENABLED`
- Entity: `PolarisBaseEntity.dropTimestamp` / `purgeTimestamp` / `toPurgeTimestamp` / `isDropped()`
- Authz: `TABLE_DROP` (covers both without-purge and with-purge paths today)

**Note:** `ENTITY_UNDROPPABLE` means “entity cannot be dropped” (protected), **not** that undrop exists.

## Requirements

| ID | Requirement | Priority |
|----|-------------|----------|
| R1 | Catalog-level enable flag; default **false** | P0 |
| R2 | Configurable hold period before auto permanent-delete eligibility | P0 |
| R3 | Soft-deleted tables invisible to normal list/load/HEAD | P0 |
| R4 | Active name **freed** after soft-delete (recreate allowed) | P0 |
| R5 | Soft-delete retains metadata location and entity payload needed for recovery | P0 |
| R6 | Soft-delete does **not** delete data or metadata files | P0 |
| R7 | Permanent delete after hold reuses existing cleanup machinery when data purge is requested | P0 |
| R8 | Privileged force permanent delete (skip hold) | P0 |
| R9 | Authz distinguishes soft-drop, undrop, force-purge, list-soft-deleted | P0 |
| R10 | Audit fields / events: actor, time, purge-not-before, original identifier | P0 |
| R11 | INTERNAL Iceberg tables only for P0; clear error for unsupported catalog types | P0 |
| R12 | Undrop if original name free; structured conflict if recreated | P1 |
| R13 | List soft-deleted (admin / privileged) | P1 |
| R14 | Views follow same model | P2 |
| R15 | Optional policy type for hold inheritance | P2 |

## Proposed design

### Lifecycle

```
DROP (soft) hold expires / FORCE_PURGE
ACTIVE ---------------------> SOFT_DELETED --------------------------> PURGED
name freed | (catalog gone;
files intact | optional file cleanup)
|
UNDROP
|
v
ACTIVE
(if name free)
```

### Configuration (catalog-scoped, additive)

| Config key | Default | Description |
|------------|---------|-------------|
| `polaris.config.table-soft-delete.enabled` | `false` | Master switch |
| `polaris.config.table-soft-delete.hold-period` | `P7D` (ISO-8601 duration) | Minimum time before auto permanent-delete eligibility |
| `polaris.config.table-soft-delete.purge-data-on-permanent-delete` | `false` or align with site policy | Whether permanent delete enqueues file cleanup |
| Realm feature flag (same pattern as other features) | off | Multi-tenant kill switch |

Naming follows existing `polaris.config.*` catalog overrides (see `DROP_WITH_PURGE_ENABLED` / `polaris.config.drop-with-purge.enabled`). Exact key names subject to community bikeshed; semantics above are normative.

### Iceberg REST behavior (when enabled)

| Client call | Behavior |
|-------------|----------|
| `DELETE .../tables/{table}` **without** purge | **Soft-delete** (R3–R6) |
| `DELETE .../tables/{table}?purge=true` | **Force permanent delete** (skip hold) if authorized; else **403**. Matches operator intent of “destroy now” and today’s drop-with-purge privilege model |
| `GET` / `HEAD` / list tables | Soft-deleted **not found** / omitted |
| Create table with same identifier | **Allowed** after soft-delete |

Engines keep using standard DROP / DROP PURGE. No Iceberg REST undrop is assumed in v1.

### Polaris-native APIs (Phase 2)

Additive endpoints under Polaris-controlled API surface (compatible evolution per [evolution docs](https://polaris.apache.org/in-dev/unreleased/evolution/)):

```http
GET /polaris/v1/{prefix}/soft-deleted-tables?namespace=...&pageToken=...
POST /polaris/v1/{prefix}/namespaces/{namespace}/tables/{table}/undrop
POST /polaris/v1/{prefix}/namespaces/{namespace}/tables/{table}/purge-now
```

Optional CLI:

```bash
polaris tables list-soft-deleted --catalog C --namespace N
polaris tables undrop --catalog C --namespace N T
polaris tables purge-now --catalog C --namespace N T
```

### Undrop rules (Phase 2)

1. Soft-deleted record exists for the catalog + original identifier (and/or entity id).
2. If no **active** entity occupies that identifier → restore: clear drop timestamps, re-index as active.
3. If name occupied → **409 Conflict** with structured error; optional `undrop-as` / alternate name is a later extension.
4. **Grants (v1 recommendation):** retain grant records across soft-delete so undrop restores access without manual re-grant (open question O1).

### Permanent delete and expiration task

An **expiration task** (admin-tool command and/or background task executor / Helm maintenance CronJob pattern):

1. Select soft-deleted entities with `now >= toPurgeTimestamp`.
2. Permanently remove catalog state (same end state as today’s hard drop).
3. If data purge configured for that entity/catalog → enqueue existing `ENTITY_CLEANUP_SCHEDULER` / `TableCleanupTaskHandler`.

Force purge / `purge-now` uses the same path immediately when authorized.

### Authz

| Operation | Privilege | Note |
|-----------|-----------|------|
| Soft-drop | `TABLE_DROP` | Today’s `DROP_TABLE_WITHOUT_PURGE` path |
| Force permanent delete / purge-now | `TABLE_DROP` + **`DROP_TABLE_WITH_PURGE`** | **Proposed:** split force-purge into a separate privilege to prevent accidental data destruction. Today both paths only require `TABLE_DROP`. |
| Undrop | **New** `TABLE_UNDROP` | Preferred over overloading `CREATE` |
| List soft-deleted | **New** `TABLE_LIST_SOFT_DELETED` or catalog-manage privilege | |

Document in RBAC docs; full authz test matrix required.

### Persistence design

#### Direction: in-place soft-delete with filtered active view

Polaris already stores "dropped" entities in `entities_active` (the name is historical; it means "not permanently purged"). The soft-delete feature reinterprets this:

- **Today:** `dropEntityIfExists()` sets `dropTimestamp`, then optionally schedules cleanup and removes the entity.
- **With soft-delete enabled:** `dropEntityIfExists()` sets `dropTimestamp` and `toPurgeTimestamp`, but **does not** schedule cleanup and **does not** remove the entity from the active index. The entity remains in `entities_active` with `isDropped() == true`, but normal resolution paths filter it out.

This requires:

1. `resolveTableLikeEntity()` and list operations to add `AND (dropTimestamp IS NULL)` (or equivalent) for the active view.
2. A new `listSoftDeletedEntities()` query path on `BasePersistence` / metastore SPI.
3. `undrop` clears `dropTimestamp` and `toPurgeTimestamp`, revalidating name uniqueness.

**Open:** Whether to add a secondary index on `(catalogId, dropTimestamp, toPurgeTimestamp)` for expiration task efficiency, or scan with filter.

#### Entity field reuse

| Field | Soft-delete use |
|-------|-----------------|
| `dropTimestamp` | Set on soft-delete (`isDropped() == true`) |
| `toPurgeTimestamp` | `dropTimestamp + hold-period` |
| `purgeTimestamp` | Set when permanent purge completes, or row removed |

**Active resolution / list:** ignore or exclude `isDropped()` entities (already partially present in resolver patterns).
**Soft-deleted store:** durable record with full table entity payload (metadata location, storage info, properties). Atomic: active name free **and** soft-deleted record committed together.

SPI contract must hold for **JDBC and NoSQL** metastores. Do **not** blindly reintroduce removed `entitiesDropped` helpers; design the API for current `BasePersistence` / atomic managers.

### Modified code paths

| Method | Change |
|--------|--------|
| `IcebergCatalogHandler.dropTableWithoutPurge()` | When soft-delete enabled: call `metaStoreManager.softDeleteEntity()` instead of `dropEntityIfExists()` |
| `IcebergCatalogHandler.dropTableWithPurge()` | Unchanged semantics (force permanent delete); may need privilege check refinement per new `DROP_TABLE_WITH_PURGE` proposal |
| `AtomicOperationMetaStoreManager.dropEntityIfExists()` | Add `softDelete` boolean or new `softDeleteEntity()` method; skip cleanup task when soft-deleting |
| `TableCleanupTaskHandler` | No change; only invoked on permanent delete |
| New: `MetaStoreManager.permanentlyDeleteEntity()` | For expiration task and force-purge; calls `dropEntityIfExists(cleanup=true)` or equivalent |

### Migration

Soft-delete is **opt-in per catalog** and default-off. Existing dropped entities in the metastore are **not** retroactively recoverable (they lack `toPurgeTimestamp` and full metadata retention). When enabled, only future `DROP` operations create soft-deleted records. The expiration task must filter to entities with `toPurgeTimestamp IS NOT NULL` to avoid accidentally re-processing historical drops.

### Interaction with table maintenance (#538)

| Job | Soft-deleted tables |
|-----|---------------------|
| Compaction / snapshot expiry | **Skip** by default (not active) |
| Orphan file removal | **Must not** treat soft-deleted table files as orphans until permanent delete |
| Drop-time file purge | Only on **permanent** delete when configured |

This boundary is required for acceptance: soft-delete without orphan-safety is a data-loss footgun.

### Events / audit

Emit events when the event listener framework is available:

- `TableSoftDeleted` - actor, catalog, identifier, entity id, `toPurgeTimestamp`
- `TableUndropped`
- `TablePermanentlyDeleted`

Optional internal properties: `polaris.soft-delete.dropped-by`, `polaris.soft-delete.reason`.

### Policy type note (R15)

A `system.table-retention` policy type (inheritable, attachable to catalog/namespace/table) is an attractive alternative to catalog-scoped config. It would define `hold_period_days`, `enable_soft_delete`, and `purge_after_hold`. This aligns with Polaris's policy framework and Floe integration. **Deferred to post-P0** to avoid blocking the core lifecycle feature on policy design.

## Alternatives considered

| Alternative | Decision |
|-------------|----------|
| **Rename into quarantine namespace** | Valid **operator workaround**, poor product: authz bypass, name collisions, still “live” tables, no standard undrop. Document as interim ops pattern only. |
| **Hard drop without purge + external ledger + `registerTable`** | Externalizes source of truth; not loadable during hold; orphan GC risk. Acceptable private ops, not core design. |
| **Storage versioning / Object Lock only** | Protects bytes, not catalog identity or Iceberg usability. Complementary. |
| **Catalog branching (#3469)** | Solves multi-table isolation, not drop hold. Do not block this RFC on it. |
| **Always soft-delete (default on)** | Breaks clients that assume DROP is hard; violates opt-in compatibility. |
| **Reserve name until hold ends** | Breaks drop-and-recreate ETL; freeing name is preferred with undrop conflict rules. |
| **Revive old `entitiesDropped` as-is** | Explicitly deferred in #1070; redesign against modern SPI. |

Start by tracing LocalIcebergCatalog.dropTable and IcebergCatalogHandler.dropTableWithoutPurge/dropTableWithPurge into dropEntityIfExists, then read PolarisBaseEntity timestamps and the ENTITY_CLEANUP_SCHEDULER/TableCleanupTaskHandler path. Compare the existing behavior with requirements R1-R15; done would require a reviewed, phased design and implementation covering soft-delete state, filtering, expiration, authorization, and recovery APIs.

Written by the indexing model from the issue text.