Non-realtime tasks can get stuck forever in waiting state after overlord restarts if their locks are revoked
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
Since 0.12.
### Description
Steps to reproduce:
1. A task of a low priority is issued. It immediately gets locks but waits for a task slot to become available in pending state.
2. Overlord is restarted.
3. After the restart, all information of tasks and their locks are restored from metadata store. However, their last states are not. Instead, all tasks are scheduled by going through `TaskQueue` as they were just issued, but with the restored locks.
4. In `TaskQueue`, it checks if tasks are ready to run by calling `Task.isReady()`. Realtime tasks always return true. Other task types such as batch or kill tasks can return false if they fail to acquire locks or their locks are revoked.
5. If their locks are revoked, `Task.isReady()` never returns true, and thus those tasks never get scheduled to `RemoteTaskRunner`.
Contributor guide
Research direction
Start with TaskQueue, Task.isReady(), and the scheduling path to RemoteTaskRunner; inspect how restored tasks and revoked locks are handled after an overlord restart. Confirm the behavior using the reproduction steps, then ensure revoked-lock tasks are no longer left unscheduled in the waiting state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100