apache / apache/polaris

Feature: Distributed Background Task Sweeper Daemon (Task GC)

Open
#4,922 3 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
2d 1h
Merged PRs (30d)
140

Description

## Problem
Polaris writes asynchronous task metadata (such as file cleanup tasks after a `DROP TABLE` operation) to the database as `TASK` entities. These tasks are scheduled to run asynchronously. However, if a node crashes, the JVM restarts, or a transient error occurs during execution, these tasks become orphaned and remain stuck in the database indefinitely.

Although the metastore layer implements CAS-based task leasing (`PolarisMetaStoreManager#loadTasks`), there is currently no background thread or daemon in Polaris that polls this endpoint to recover and execute orphaned tasks.

## Proposal
Introduce a distributed background task sweeper daemon (`PolarisTaskSweeper`) to periodically lease, run, and clean up stuck tasks across all active realms.

### Design Highlights
* **CAS Leasing:** Periodically calls `loadTasks` which uses internal CAS updates (`LAST_ATTEMPT_START_TIME`, `ATTEMPT_COUNT`) to lease tasks. Multiple Polaris nodes can safely sweep concurrently without double-executing tasks.
* **Async Processing:** Claims are processed asynchronously on the shared `task-executor` thread pool to avoid blocking the scheduler thread during long cloud storage cleanups.
* **Observability:** Fires standard `BEFORE_ATTEMPT_TASK` and `AFTER_ATTEMPT_TASK` events so sweeps are visible to audit loggers and OpenTelemetry listeners.
* **Thundering Herd Mitigation:** Applies a random jitter (up to 20% of the interval) before runs so multi-node deployments do not hit the database simultaneously.
* **Configuration:** Exposes `polaris.tasks.sweeper.enabled` (default: `true`) and `polaris.tasks.sweeper.interval` (default: `1m`) configuration properties.

I have a fully tested local implementation of this daemon (including unit tests and documentation updates) ready on a branch and can submit a PR.

Contributor guide

Open the contributing guide

Research direction

Start by reading PolarisMetaStoreManager#loadTasks and the existing task-executor and task event flows. Then trace how configuration properties are defined and consumed. Done means a distributed PolarisTaskSweeper leases and asynchronously processes orphaned tasks with jitter, configuration, observability events, unit tests, and documentation updates as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.