apache / apache/ignite

Apache Ignite Rebalancing

Open
#12,477 3 comments 0 reactions 0 assignees View on GitHub
ignite-3
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

I was working on Apache ignitev2.17 and started upgrading to v3.0 . I have 3 server nodes and .net api as a client. I created one zone with Replicas 2 and scaledown = 5 and created a table belong to this zone, then I made one of these nodes offline. the cluster keeps working fine (I can check cluster status, zones,..) but if I do query to data, I got issue like this

```
SELECT * from t_read_status limit 1;
SQL query execution error
Mandatory nodes was excluded from mapping: [Node3]
```

or

```
select count(*) from t_read_status;
SQL query execution error
The primary replica await timed out [replicationGroupId=17_part_1, referenceTimestamp=HybridTimestamp [physical=2025-10-28 11:47:20:016 +0000, logical=0, composite=115451628094488576], currentLease=Lease [leaseholder=Node2, leaseholderId=486d7bc8-b03e-4c0f-91a8-76a6bb0dc243, accepted=false, startTime=HybridTimestamp [physical=2025-10-28 11:47:14:874 +0000, logical=95, composite=115451627757502559], expirationTime=HybridTimestamp [physical=2025-10-28 11:49:14:874 +0000, logical=0, composite=115451635621822464], prolongable=true, proposedCandidate=null, replicationGroupId=17_part_1]]
```
and I checked the online node logs and found
```
[WARNING][%Node1%replica-4][ReplicaManager] Failed to process the lease granted message
```

then I tried to change the cluster config of replication.leaseExpirationInterval to 30000 then I did the test again I have the same result but when I check the node logs found
```
RebalanceRaftGroupEventsListener] Going to retry rebalance [attemptNo=16908, partId=20_part_18]
[ERROR][%Node1%JRaft-Request-Processor-8][ReplicatorGroupImpl] Fail to check replicator connection to peer=Node2, replicatorType=Follower.
[RebalanceRaftGroupEventsListener] Number of retries for rebalance exceeded the threshold [partId=17_part_18, threshold=10]
```

I mentioned some of the logs , and I am lost don't know why the cluster (raft) can't do rebalancing

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the three-node setup with replication.leaseExpirationInterval changed, then inspect the RebalanceRaftGroupEventsListener, ReplicaManager, and ReplicatorGroupImpl log entries cited in the report. Done means identifying why rebalancing fails after Node3 goes offline and documenting or correcting the behavior so queries continue working with the configured replica count.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.