apache / apache/druid

A more granular method of retaining recently completed tasks

Open
#8,696 0 comments 0 reactions 0 assignees View on GitHub
Area - Metadata Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

The sys.tasks table has an existing setting "druid.indexer.storage.recentlyFinishedThreshold" that dictates how how long tasks are kept in history (for viewing within the console or API).

In cases where there are an unusually large number of tasks that are tracked within the retention period shown above, then there is the possibility of causing significant performance issues, especially when trying to work with a large number of failed tasks.

I would like to propose two changes:

1. add a "druid.indexer.storage.recentlyFinishedThresholdPeriod" and "druid.indexer.storage.recentlyFinishedThresholdCount" setting so that users could specify a threshold in terms of time and raw count. The server would be expected to honor whichever of the two settings triggers first.

2. In addition to the new, more granular settings, allow both of those settings to be applied to tasks based on the task status (e.g. SUCCESS, FAILED, etc.) and include a prioritization scheme for that. This way, One could, for example, discard tasks that succeeded more quickly than those that failed. Additionally, it might be useful to be able to throttle failed tasks by sampling, perhaps.

The reason for providing such fine-grained control over these settings is that in essence, a large number of failed tasks could, in the worst case, cause a self-inflicted denial of service or severe enough degradation of system performance that the effect would be similar and compound the difficulty and time taken to troubleshoot the system (when the reason for the glut of tasks is due to failures).

Contributor guide

Open the contributing guide

Research direction

Start by locating the sys.tasks retention implementation and the existing druid.indexer.storage.recentlyFinishedThreshold setting. Review how recently finished tasks are retained and exposed through the console or API. Done would require an agreed design and implementation for time- and count-based limits, status-specific prioritization, and any failed-task sampling behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.