[multistage] [bug] Sort operator issue when # of pinot servers is greater 1
Open
multi-stage
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
StagePlan is sent to all available intermediate stage servers. When # of intermediate stage servers is greater than 1, only first server will receive leaf server data.
The other servers will spin operator waiting on mailbox.
Correctness: if any of first server returns first, we will just return error.
Wasted CPU: All servers except 1, will spin CPU on waiting something they don’t need. With non-blocking fix(https://github.com/apache/pinot/issues/9615#issuecomment-1282911839), this will become a memory “leak” problem.
HotSpot: First server gets way more load than the other server
Should be related to https://github.com/apache/pinot/issues/9611
Contributor guide
Assessment
This issue has not been assessed yet.