apache / apache/ignite

Apache Ignite data rebalancing problem (Will not own partition (there are owners to rebalance from))

Open
#10,859 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

I updated Apache Ignite to version 2.15.0
Everything is fine when I launch the first node, it works, the client connects to it.
But when I launch the second node, an error occurs at the balancing stage, as far as I understand
Messages like this start to appear in the logs:
[2023-07-25T12:50:39,248][DEBUG][exchange-worker-#50][GridDhtPartitionTopologyImpl] Will not own partition (there are owners to rebalance from) [grp=CustomCacheItem, p=3, owners = [TcpDiscoveryNode [id=2cb9ce8d-2025-456d-82a6-c761ef7a98b7, consistentId=1NodeHostname, addrs=ArrayList [20.20.28.215, 127.0.0.1], sockAddrs=HashSet [1NodeHostname/20.20.28.215:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1690278575122, loc=false, ver=2.15.0#20230425-sha1:f98f7f35, isClient=false]]]

Then there are these errors appear:
[2023-07-25T12:50:55,935][ERROR][tcp-disco-msg-worker-[2cb9ce8d 20.20.28.215:47500]-#2-#44][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=partition-exchanger, threadName=exchange-worker-#50, blockedFor=16s]

And second node is shutting down
It seems that the problem occurs during rebalancing. But what exactly is the problem?

Contributor guide

Open the contributing guide

Research direction

Start with the reported rebalancing logs, especially GridDhtPartitionTopologyImpl and the partition-exchanger blocked-thread message, while reproducing the two-node startup described here. Trace why the second node shuts down during partition ownership transfer; done means identifying a concrete cause and documenting a verified remediation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.