HttpRemoteTaskRunner: Overlord stalling on huge backlog
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
`HttpRemoteTaskRunner.pendingTasksExecutionLoop()` holds the single `statusLock` monitor while iterating `pendingTaskIds`, and for **every** pending task it calls `findWorkerToRunTask(Task)`, which rebuilds a full immutable snapshot of all workers:
```java
// findWorkerToRunTask(Task)
return strategy.findWorkerForTask(
config,
ImmutableMap.copyOf(getWorkersEligibleToRunTasks()), // rebuilt on every call
task
);
```
`getWorkersEligibleToRunTasks()` filters and transforms the whole `workers` map, and each `WorkerHolder.toImmutable()` reconstructs that worker's announced-task set via `ImmutableWorkerInfo.fromWorkerAnnouncements(...)`.
```
O(pendingTasks × workers × tasksAnnouncedPerWorker)
```
…and the entire pass is executed while holding `statusLock`.
At small backlogs this is invisible. Under a large pending backlog with the cluster at/near capacity, a single loop pass holds `statusLock` for many seconds to minutes. Because `statusLock` is also required by `run()` (new task submission), `taskComplete()` / status updates, and the worker-sync path, the Overlord effectively freezes.
Restarting the Overlord does not recover it: the active task set is persisted in metadata and reloaded via `syncFromStorage` on startup, so `pendingTaskIds` is immediately large again and the loop re-enters the same lock-holding scan.
Observed on Druid 33.0.0 (`httpRemote` task runner).
Thread-dump signature at the stall:
- One `hrtr-pending-tasks-runner-*` thread is `RUNNABLE`, holding `statusLock`, deep in `ImmutableWorkerInfo.fromWorkerAnnouncements` → `WorkerHolder.toImmutable` → `getWorkersEligibleToRunTasks` → `findWorkerToRunTask` → `pendingTasksExecutionLoop`.
- The other pending-task-runner threads are idle in `statusLock.wait()`.
- `TaskQueue-Manager` is `BLOCKED` on the same monitor in `HttpRemoteTaskRunner.run()`, while holding the `TaskQueue` giant lock.
- Many Jetty `qtp-*` handler threads are parked on the `TaskQueue` lock in `OverlordResource.taskPost → TaskQueue.add`.
### Motivation
**Use case:** any Overlord using the `httpRemote` task runner that can accumulate a large pending-task backlog while workers are saturated
**Why the change is beneficial:**
- No behavior change for correctness: within a single synchronized pass the loop reserves at most one task (it `break`s right after `workersWithUnacknowledgedTask.putIfAbsent`), so the eligible-worker set is invariant across the inner loop — recomputing it per task produces identical results.
**Proposed fix:**
**Compute the eligible-worker snapshot once per pass**, not once per pending task. Build it just inside `synchronized (statusLock)` before iterating `pendingTaskIds`, and pass it into an overload `findWorkerToRunTask(Task, ImmutableMap eligibleWorkers)`. This drops the dominant `fromWorkerAnnouncements` rebuild cost from O(pendingTasks × workers × tasksPerWorker) to O(workers × tasksPerWorker) per pass.
Sketch:
```java
synchronized (statusLock) {
final ImmutableMap eligibleWorkers =
ImmutableMap.copyOf(getWorkersEligibleToRunTasks()); // once per pass
Iterator iter = pendingTaskIds.iterator();
while (iter.hasNext()) {
...
immutableWorker = findWorkerToRunTask(ti.getTask(), eligibleWorkers); // no per-task rebuild
...
}
...
}
```
### Affected Version
Reproduced on 33.0.0; code path unchanged on `master`.
---
Used some AI to structure the issue.
Contributor guide
Research direction
Start at HttpRemoteTaskRunner.pendingTasksExecutionLoop() and trace findWorkerToRunTask(Task), getWorkersEligibleToRunTasks(), and WorkerHolder.toImmutable() to confirm where the repeated snapshot work occurs. Done means the eligible-worker snapshot is built once per synchronized pass, reused for pending tasks, and the existing scheduling behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100