apache / apache/druid

HttpRemoteTaskRunner: Overlord stalling on huge backlog

Open
#20,295 1 comment 0 reactions 0 assignees View on GitHub
Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

`HttpRemoteTaskRunner.pendingTasksExecutionLoop()` holds the single `statusLock` monitor while iterating `pendingTaskIds`, and for **every** pending task it calls `findWorkerToRunTask(Task)`, which rebuilds a full immutable snapshot of all workers:

```java
// findWorkerToRunTask(Task)
return strategy.findWorkerForTask(
config,
ImmutableMap.copyOf(getWorkersEligibleToRunTasks()), // rebuilt on every call
task
);
```

`getWorkersEligibleToRunTasks()` filters and transforms the whole `workers` map, and each `WorkerHolder.toImmutable()` reconstructs that worker's announced-task set via `ImmutableWorkerInfo.fromWorkerAnnouncements(...)`.

```
O(pendingTasks × workers × tasksAnnouncedPerWorker)
```

…and the entire pass is executed while holding `statusLock`.

At small backlogs this is invisible. Under a large pending backlog with the cluster at/near capacity, a single loop pass holds `statusLock` for many seconds to minutes. Because `statusLock` is also required by `run()` (new task submission), `taskComplete()` / status updates, and the worker-sync path, the Overlord effectively freezes.

Restarting the Overlord does not recover it: the active task set is persisted in metadata and reloaded via `syncFromStorage` on startup, so `pendingTaskIds` is immediately large again and the loop re-enters the same lock-holding scan.

Observed on Druid 33.0.0 (`httpRemote` task runner).

Thread-dump signature at the stall:
- One `hrtr-pending-tasks-runner-*` thread is `RUNNABLE`, holding `statusLock`, deep in `ImmutableWorkerInfo.fromWorkerAnnouncements` → `WorkerHolder.toImmutable` → `getWorkersEligibleToRunTasks` → `findWorkerToRunTask` → `pendingTasksExecutionLoop`.
- The other pending-task-runner threads are idle in `statusLock.wait()`.
- `TaskQueue-Manager` is `BLOCKED` on the same monitor in `HttpRemoteTaskRunner.run()`, while holding the `TaskQueue` giant lock.
- Many Jetty `qtp-*` handler threads are parked on the `TaskQueue` lock in `OverlordResource.taskPost → TaskQueue.add`.

### Motivation

**Use case:** any Overlord using the `httpRemote` task runner that can accumulate a large pending-task backlog while workers are saturated

**Why the change is beneficial:**
- No behavior change for correctness: within a single synchronized pass the loop reserves at most one task (it `break`s right after `workersWithUnacknowledgedTask.putIfAbsent`), so the eligible-worker set is invariant across the inner loop — recomputing it per task produces identical results.

**Proposed fix:**

**Compute the eligible-worker snapshot once per pass**, not once per pending task. Build it just inside `synchronized (statusLock)` before iterating `pendingTaskIds`, and pass it into an overload `findWorkerToRunTask(Task, ImmutableMap eligibleWorkers)`. This drops the dominant `fromWorkerAnnouncements` rebuild cost from O(pendingTasks × workers × tasksPerWorker) to O(workers × tasksPerWorker) per pass.

Sketch:

```java
synchronized (statusLock) {
final ImmutableMap eligibleWorkers =
ImmutableMap.copyOf(getWorkersEligibleToRunTasks()); // once per pass

Iterator iter = pendingTaskIds.iterator();
while (iter.hasNext()) {
...
immutableWorker = findWorkerToRunTask(ti.getTask(), eligibleWorkers); // no per-task rebuild
...
}
...
}
```

### Affected Version

Reproduced on 33.0.0; code path unchanged on `master`.

---

Used some AI to structure the issue.

Contributor guide

Open the contributing guide

Research direction

Start at HttpRemoteTaskRunner.pendingTasksExecutionLoop() and trace findWorkerToRunTask(Task), getWorkersEligibleToRunTasks(), and WorkerHolder.toImmutable() to confirm where the repeated snapshot work occurs. Done means the eligible-worker snapshot is built once per synchronized pass, reused for pending tasks, and the existing scheduling behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.