apache / apache/polaris

Task handling is incomplete

Open
#774 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

Polaris uses some asynchronously executed tasks to run operations for table and manifest file cleanup. Those tasks are potentially executed in a separate thread in the same JVM. There is however no guarantee that those tasks will eventually run for multiple reasons:

* Tasks (e.g. via `org.apache.polaris.service.catalog.BasePolarisCatalog#dropTable`) are triggered _after_ the fact.
* Although tasks are persisted, there is no mechanism to pick up tasks that did not start or did not finish ("long lasting" failures, JVM terminates).

Overall this means that for example a "drop table with purge" returns a successful result to the user, the actual purge may never ever happen.

### To Reproduce

_No response_

### Actual Behavior

_No response_

### Expected Behavior

_No response_

### Additional context

_No response_

### System information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at org.apache.polaris.service.catalog.BasePolarisCatalog#dropTable and trace how cleanup tasks are triggered and persisted. Review how tasks that never start or do not finish are handled; done should ensure operations such as table drops with purge cannot report success while cleanup is permanently abandoned.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.