apache / apache/helix

WAGED Instance Capacity NPE prevents topstate handoff during rebalance failures

Open
#3,009 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
504
Forks
255
Avg merge
17d 47m
Merged PRs (30d)
2

Description

### Describe the bug
During rebalance failures, topState handoff will still occur among nodes that already have assigned replicas. This is because WAGED leverages the the previous best possible assignment when calculating the best possible partition state. However, if the cluster is in a state of rebalance failures and a node is removed, then WAGED will look at the previous best possible which contains an instance that is no longer in the cluster. This stale instance will trigger an NPE when we attempt to check its instance capacity as it no longer exists in the instanceCapacityMap. NPEs are part of the failure_type_to_propagate which cause the bestPossibleCalcStage to fully crash, meaning we do not calculate new states based on the previous assignment so no topState handoff will occur. This can cause a partition to go leaderless until rebalance failures are fixed and a new best possible is generated.

### To Reproduce
1. Trigger waged rebalance failures
a. This can be done by adding more resources than can be assigned for cluster's capacity
2. Remove an instance from the cluster
3. Next time a node goes down, topState handoff will not occur

### Expected behavior
Partition states are still calculated during rebalance failures using the old assignment so that top state handoff can still occur

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing WAGED's bestPossibleCalcStage during rebalance failures, focusing on how the previous best possible assignment is checked against instanceCapacityMap after an instance is removed. Reproduce the sequence by causing capacity-exceeding rebalance failures, removing an instance, and taking a node down. Done means partition states are still calculated and topState handoff occurs without an NPE.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.