apache / apache/pinot

Reduce number of Helix threads in Pinot server

Open
#7,748 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

In numerous profiles collected from different deployments, the number of `HelixTaskExecutor-message_handle_thread` is always 42, even in small deployments with 4 cores and 4GB heap per server. This number seems far too high and does not depend on the number of cores available, so should be reduced to depend on the number of cores available to the server.
Screenshot 2021-11-11 at 14 41 45

Contributor guide

Open the contributing guide

Research direction

The issue identifies HelixTaskExecutor-message_handle_thread in the Pinot server; start by locating where this thread count is configured or created. Compare the current fixed count with the server's available cores, then verify on a small deployment that the number of Helix threads scales with cores without affecting server operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.