apache / apache/polaris

Cascading Asynchronous Catalog Deletion

Open
#3,008 3 comments 0 reactions 0 assignees View on GitHub
enhancement stale
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Is your feature request related to a problem? Please describe.

On Slack, a user mentioned that they were having trouble cleaning up an unused catalog. They had deleted tables and namespaces, but forgot to delete catalog roles. They got into this state because they change their underlying object store (GCS) and they wanted to just start with a fresh new catalog.

### Describe the solution you'd like

An extension to the Polaris Management REST API to allow for a catalog and all its related content to be deleted in the background.
As @snazy noted:
1. This must also be resumable because the metastore DB might stall or fail.
2. The catalog must not be usable for others while it is being purged. This requires a catalog-state-management and a corresponding catalog-state-machine.

### Describe alternatives you've considered

1. Delete all the catalog entities one-by-one

### Additional context

1. Eric Maynard mentioned that there was some vestigial code related to CASCADE (e.g. CLEANUP_ON_CATALOG_DROP).
2. The maintenance service project / asynchronous task execution is related to this.
3. As @flyrain noted, we will also need to consider polices that attached to entities in different catalogs.

Contributor guide

Open the contributing guide

Research direction

Start with the Polaris Management REST API and the maintenance service or asynchronous task execution project mentioned in the issue. Trace how catalog state management and the catalog state machine would coordinate a background purge that can resume after metastore failures and prevent use during deletion. Done means related catalog content, including attached policies, is safely purged with resumability and the catalog remains unavailable while the work runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.