apache / apache/druid

Delegate cost computation in CostBalancerStrategy to the common ForkJoinPool and remove balancerComputeThreads config

Open
#7,143 0 comments 0 reactions 0 assignees View on GitHub
Area - Segment Balancing/Coordination
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

I don't see a point of using custom thread pool for `CostBalancerStrategy`, because it delegates to the pool "pure", non-blocking `computeCost()` calls. When the logic to be parallelized is non-blocking, it's a job for the common `ForkJoinPool` via parallel Streams.

This will likely also improve scalability because instead of wrapping each `computeCost()` call as a separate task for a thread pool (this might be the reason for poor scalability - see #3275), parallel `Stream` won't execute more slices than there are cores in the system.

In the transition period, the `balancerComputeThreads` may be ignored and a warning or an alert is issued if this option is found in the config.

FYI @himadrisingh001 @xvrl @fjy - contributors to the original change that introduced parallelization (#399).

Contributor guide

Open the contributing guide

Research direction

Start by reading CostBalancerStrategy and its computeCost() parallelization, then trace where balancerComputeThreads is read and applied. Replace the custom pool usage with the common ForkJoinPool through parallel Streams, and verify that the legacy configuration is ignored with a warning or alert when present.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.