Feature: Distributed Background Task Sweeper Daemon (Task GC)
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 140
Description
## Problem
Polaris writes asynchronous task metadata (such as file cleanup tasks after a `DROP TABLE` operation) to the database as `TASK` entities. These tasks are scheduled to run asynchronously. However, if a node crashes, the JVM restarts, or a transient error occurs during execution, these tasks become orphaned and remain stuck in the database indefinitely.
Although the metastore layer implements CAS-based task leasing (`PolarisMetaStoreManager#loadTasks`), there is currently no background thread or daemon in Polaris that polls this endpoint to recover and execute orphaned tasks.
## Proposal
Introduce a distributed background task sweeper daemon (`PolarisTaskSweeper`) to periodically lease, run, and clean up stuck tasks across all active realms.
### Design Highlights
* **CAS Leasing:** Periodically calls `loadTasks` which uses internal CAS updates (`LAST_ATTEMPT_START_TIME`, `ATTEMPT_COUNT`) to lease tasks. Multiple Polaris nodes can safely sweep concurrently without double-executing tasks.
* **Async Processing:** Claims are processed asynchronously on the shared `task-executor` thread pool to avoid blocking the scheduler thread during long cloud storage cleanups.
* **Observability:** Fires standard `BEFORE_ATTEMPT_TASK` and `AFTER_ATTEMPT_TASK` events so sweeps are visible to audit loggers and OpenTelemetry listeners.
* **Thundering Herd Mitigation:** Applies a random jitter (up to 20% of the interval) before runs so multi-node deployments do not hit the database simultaneously.
* **Configuration:** Exposes `polaris.tasks.sweeper.enabled` (default: `true`) and `polaris.tasks.sweeper.interval` (default: `1m`) configuration properties.
I have a fully tested local implementation of this daemon (including unit tests and documentation updates) ready on a branch and can submit a PR.
Contributor guide
Research direction
Start by reading PolarisMetaStoreManager#loadTasks and the existing task-executor and task event flows. Then trace how configuration properties are defined and consumed. Done means a distributed PolarisTaskSweeper leases and asynchronously processes orphaned tasks with jitter, configuration, observability events, unit tests, and documentation updates as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100