Apache Ignite data rebalancing problem (Will not own partition (there are owners to rebalance from))
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
I updated Apache Ignite to version 2.15.0
Everything is fine when I launch the first node, it works, the client connects to it.
But when I launch the second node, an error occurs at the balancing stage, as far as I understand
Messages like this start to appear in the logs:
[2023-07-25T12:50:39,248][DEBUG][exchange-worker-#50][GridDhtPartitionTopologyImpl] Will not own partition (there are owners to rebalance from) [grp=CustomCacheItem, p=3, owners = [TcpDiscoveryNode [id=2cb9ce8d-2025-456d-82a6-c761ef7a98b7, consistentId=1NodeHostname, addrs=ArrayList [20.20.28.215, 127.0.0.1], sockAddrs=HashSet [1NodeHostname/20.20.28.215:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1690278575122, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, isClient=false]]]
Then there are these errors appear:
[2023-07-25T12:50:55,935][ERROR][tcp-disco-msg-worker-[2cb9ce8d 20.20.28.215:47500]-#2-#44][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=partition-exchanger, threadName=exchange-worker-#50, blockedFor=16s]
And second node is shutting down
It seems that the problem occurs during rebalancing. But what exactly is the problem?
Contributor guide
Research direction
Start with the reported rebalancing logs, especially GridDhtPartitionTopologyImpl and the partition-exchanger blocked-thread message, while reproducing the two-node startup described here. Trace why the second node shuts down during partition ownership transfer; done means identifying a concrete cause and documenting a verified remediation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100