apache / apache/druid

Non-realtime tasks can get stuck forever in waiting state after overlord restarts if their locks are revoked

Open
#10,251 0 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Since 0.12.

### Description

Steps to reproduce:

1. A task of a low priority is issued. It immediately gets locks but waits for a task slot to become available in pending state.
2. Overlord is restarted.
3. After the restart, all information of tasks and their locks are restored from metadata store. However, their last states are not. Instead, all tasks are scheduled by going through `TaskQueue` as they were just issued, but with the restored locks.
4. In `TaskQueue`, it checks if tasks are ready to run by calling `Task.isReady()`. Realtime tasks always return true. Other task types such as batch or kill tasks can return false if they fail to acquire locks or their locks are revoked.
5. If their locks are revoked, `Task.isReady()` never returns true, and thus those tasks never get scheduled to `RemoteTaskRunner`.

Contributor guide

Open the contributing guide

Research direction

Start with TaskQueue, Task.isReady(), and the scheduling path to RemoteTaskRunner; inspect how restored tasks and revoked locks are handled after an overlord restart. Confirm the behavior using the reproduction steps, then ensure revoked-lock tasks are no longer left unscheduled in the waiting state.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.