Services not being re-deployed on topology change

Hey guys!

We discovered that when a node is re-deploying services due to multiple nodes simultaneously leaving the cluster, services are occasionally not re-deployed as expected. This can be reproduced by running the following two classes together, in separate processes. I was able to reproduce this issue once every 2-3 tries.

Node1And2.java

```
public class Node1And2 {
public static void main(String[] args) throws InterruptedException {
Ignite node1 = Ignition.start();
Ignite node2 = Ignition.start(new IgniteConfiguration().setIgniteInstanceName("node2"));

IgniteCluster cluster = node1.cluster();
UUID node1ID = cluster.localNode().id();
UUID node2ID = node2.cluster().localNode().id();

while (cluster.topology(cluster.topologyVersion()).size() != 3) {
Thread.sleep(10);
}

IgniteServices services = node1.services();
for (int i = 0; i < 10; i++) services.deployClusterSingleton("service" + i, new ServiceImpl());

Map> servicesDeployedOnNodes = services.serviceDescriptors().stream().collect(Collectors.groupingBy(serviceDescriptor -> serviceDescriptor.topologySnapshot().entrySet().stream().filter(entry -> entry.getValue() != 0).findFirst().get().getKey(), Collectors.mapping(ServiceDescriptor::name, Collectors.toList())));
System.out.println("Services deployed on node1: " + servicesDeployedOnNodes.get(node1ID));
System.out.println("Services deployed on node2: " + servicesDeployedOnNodes.get(node2ID));

IgniteAtomicLong isServiceDeploymentComplete = node1.atomicLong("isServiceDeploymentComplete", 0, true);
isServiceDeploymentComplete.getAndSet(1);

System.out.println("Now sleeping for 10s.");
Thread.sleep(10 * 1000);

System.exit(0);
}
}
```

Node3.java

```
public class Node3 {
public static void main(String[] args) throws InterruptedException {
Ignite ignite = Ignition.start();
UUID localNodeID = ignite.cluster().localNode().id();

IgniteAtomicLong isServiceDeploymentComplete = ignite.atomicLong("isServiceDeploymentComplete", 0, true);
while (isServiceDeploymentComplete.get() != 1) {
Thread.sleep(10);
}

IgniteServices services = ignite.services();
List servicesDeployedOnLocalNode = services.serviceDescriptors().stream().filter(serviceDescriptor -> serviceDescriptor.topologySnapshot().containsKey(localNodeID) && serviceDescriptor.topologySnapshot().get(localNodeID) != 0).map(ServiceDescriptor::name).collect(Collectors.toList());
System.out.println("Services deployed on node3: " + servicesDeployedOnLocalNode);

System.out.println("Now sleeping for 20s.");
Thread.sleep(20 * 1000);

servicesDeployedOnLocalNode = services.serviceDescriptors().stream().filter(serviceDescriptor -> serviceDescriptor.topologySnapshot().get(localNodeID) != 0).map(ServiceDescriptor::name).collect(Collectors.toList());
System.out.println("Services deployed on node3: " + servicesDeployedOnLocalNode + " (" + servicesDeployedOnLocalNode.size() + " total)");
}
}
```

`ServiceImpl` is a bare minimum implementation of `Service`.

ServiceImpl.java

```
public class ServiceImpl implements Service {
}
```

After the process running `Node1And2` stops and the two nodes leave the cluster, we expected all 10 services to get re-deployed on the third node. However, occasionally, we observed that not all services get re-deployed on the third node. Here's the output of both processes in such a case.

Output for Node1And2.java

```
[12:52:18] (wrn) Default Spring XML file not found (is IGNITE_HOME set?): config/default-config.xml
Sep 03, 2025 12:52:19 PM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Failed to resolve default logging config file: config/java.util.logging.properties
[12:52:19] __________ ________________
[12:52:19] / _/ ___/ |/ / _/_ __/ __/
[12:52:19] _/ // (7 7 // / / / / _/
[12:52:19] /___/\___/_/|_/___/ /_/ /x___/
[12:52:19]
[12:52:19] ver. 2.17.0#20250209-sha1:d53d4540
[12:52:19] 2025 Copyright(C) Apache Software Foundation
[12:52:19]
[12:52:19] Ignite documentation: https://ignite.apache.org
[12:52:19]
[12:52:19] Quiet mode.
[12:52:19] ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[12:52:19] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[12:52:19]
[12:52:19] OS: Windows 11 10.0 amd64
[12:52:19] VM information: OpenJDK Runtime Environment 11.0.22+7-adhoc..jdk11u OpenLogic OpenJDK 64-Bit Server VM 11.0.22+7-adhoc..jdk11u
[12:52:19] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
[12:52:20] Configured plugins:
[12:52:20] ^-- None
[12:52:20]
[12:52:20] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
[12:52:20] Initial heap size is 384MB (should be no less than 512MB, use -Xms512m -Xmx512m).
[12:52:20] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[12:52:25] Data Regions Started: 3
[12:52:25]
[12:52:25] ^-- sysMemPlc region [type=internal, persistence=false, lazyAlloc=false,
[12:52:25] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=40MB]
[12:52:25] ^-- default region [type=default, persistence=false, lazyAlloc=true,
[12:52:25] ... initCfg=256MB, maxCfg=4898MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
[12:52:25] ^-- volatileDsMemPlc region [type=user, persistence=false, lazyAlloc=true,
[12:52:25] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
[12:52:25] Security status [authentication=off, sandbox=off, tls/ssl=off]
[12:52:25] Performance suggestions for grid (fix if possible)
[12:52:25] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[12:52:25] ^-- Specify JVM heap max size (add '-Xmx[g|G|m|M|k|K]' to JVM options)
[12:52:25] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options)
[12:52:25] ^-- Disable assertions (remove '-ea' from JVM options)
[12:52:25] Refer to this page for more performance suggestions: https://ignite.apache.org/docs/latest/perf-and-troubleshooting/memory-tuning
[12:52:25]
[12:52:25]
[12:52:25] Ignite node started OK (id=a6e3015d)
[12:52:25] Topology snapshot [ver=1, locNode=a6e3015d, servers=1, clients=0, state=ACTIVE, CPUs=8, offheap=4.8GB, heap=6.0GB]
[12:52:25] ^-- Baseline [id=0, size=1, online=1, offline=0]
[12:52:25] __________ ________________
[12:52:25] / _/ ___/ |/ / _/_ __/ __/
[12:52:25] _/ // (7 7 // / / / / _/
[12:52:25] /___/\___/_/|_/___/ /_/ /x___/
[12:52:25]
[12:52:25] ver. 2.17.0#20250209-sha1:d53d4540
[12:52:25] 2025 Copyright(C) Apache Software Foundation
[12:52:25]
[12:52:25] Ignite documentation: https://ignite.apache.org
[12:52:25]
[12:52:25] Quiet mode.
[12:52:25] ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[12:52:25] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[12:52:25]
[12:52:25] OS: Windows 11 10.0 amd64
[12:52:25] VM information: OpenJDK Runtime Environment 11.0.22+7-adhoc..jdk11u OpenLogic OpenJDK 64-Bit Server VM 11.0.22+7-adhoc..jdk11u
[12:52:25] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
[12:52:25] Configured plugins:
[12:52:25] ^-- None
[12:52:25]
[12:52:25] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
[12:52:25] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[12:52:26] Initial heap size is 384MB (should be no less than 512MB, use -Xms512m -Xmx512m).
[12:52:29] Joining node doesn't have stored group keys [node=00dc7f4a-d0e4-4391-bd04-28b9ef934050]
[12:52:29] Topology snapshot [ver=2, locNode=a6e3015d, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=4.8GB, heap=6.0GB]
[12:52:29] ^-- Baseline [id=0, size=2, online=2, offline=0]
[12:52:29] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=22244MB, available=24492MB]
[12:52:29] Data Regions Started: 3
[12:52:29]
[12:52:29] ^-- sysMemPlc region [type=internal, persistence=false, lazyAlloc=false,
[12:52:29] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=40MB]
[12:52:29] ^-- default region [type=default, persistence=false, lazyAlloc=true,
[12:52:29] ... initCfg=256MB, maxCfg=4898MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
[12:52:29] ^-- volatileDsMemPlc region [type=user, persistence=false, lazyAlloc=true,
[12:52:29] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
[12:52:30] Security status [authentication=off, sandbox=off, tls/ssl=off]
[12:52:30] Performance suggestions for grid 'node2' (fix if possible)
[12:52:30] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[12:52:30] ^-- Specify JVM heap max size (add '-Xmx[g|G|m|M|k|K]' to JVM options)
[12:52:30] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options)
[12:52:30] ^-- Disable assertions (remove '-ea' from JVM options)
[12:52:30] Refer to this page for more performance suggestions: https://ignite.apache.org/docs/latest/perf-and-troubleshooting/memory-tuning
[12:52:30]
[12:52:30]
[12:52:30] Ignite node started OK (id=00dc7f4a, instance name=node2)
[12:52:30] Topology snapshot [ver=2, locNode=00dc7f4a, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=4.8GB, heap=6.0GB]
[12:52:30] ^-- Baseline [id=0, size=2, online=2, offline=0]
[12:52:31] Joining node doesn't have stored group keys [node=0ddda712-bb41-4814-9124-86e010ce60c9]
[12:52:31] Topology snapshot [ver=3, locNode=00dc7f4a, servers=3, clients=0, state=ACTIVE, CPUs=8, offheap=9.6GB, heap=12.0GB]
[12:52:31] ^-- Baseline [id=0, size=3, online=3, offline=0]
[12:52:31] Topology snapshot [ver=3, locNode=a6e3015d, servers=3, clients=0, state=ACTIVE, CPUs=8, offheap=9.6GB, heap=12.0GB]
[12:52:31] ^-- Baseline [id=0, size=3, online=3, offline=0]
Services deployed on node1: [myService6, myService5, myService3, myService0, myService9]
Services deployed on node2: [myService1]
Now sleeping for 10s.
[12:52:43] Ignite node stopped OK [name=node2, uptime=00:00:12.957]
[12:52:43] Ignite node stopped OK [uptime=00:00:17.336]

Process finished with exit code 0
```

Output for Node3.java

```
[12:52:24] (wrn) Default Spring XML file not found (is IGNITE_HOME set?): config/default-config.xml
Sep 03, 2025 12:52:24 PM org.apache.ignite.logger.java.JavaLogger warning
WARNING: Failed to resolve default logging config file: config/java.util.logging.properties
[12:52:25] __________ ________________
[12:52:25] / _/ ___/ |/ / _/_ __/ __/
[12:52:25] _/ // (7 7 // / / / / _/
[12:52:25] /___/\___/_/|_/___/ /_/ /x___/
[12:52:25]
[12:52:25] ver. 2.17.0#20250209-sha1:d53d4540
[12:52:25] 2025 Copyright(C) Apache Software Foundation
[12:52:25]
[12:52:25] Ignite documentation: https://ignite.apache.org
[12:52:25]
[12:52:25] Quiet mode.
[12:52:25] ^-- Logging by 'JavaLogger [quiet=true, config=null]'
[12:52:25] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[12:52:25]
[12:52:25] OS: Windows 11 10.0 amd64
[12:52:25] VM information: OpenJDK Runtime Environment 11.0.22+7-adhoc..jdk11u OpenLogic OpenJDK 64-Bit Server VM 11.0.22+7-adhoc..jdk11u
[12:52:25] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
[12:52:26] Configured plugins:
[12:52:26] ^-- None
[12:52:26]
[12:52:26] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
[12:52:26] Initial heap size is 384MB (should be no less than 512MB, use -Xms512m -Xmx512m).
[12:52:26] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[12:52:31] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=33367MB, available=24492MB]
[12:52:31] Data Regions Started: 3
[12:52:31]
[12:52:31] ^-- sysMemPlc region [type=internal, persistence=false, lazyAlloc=false,
[12:52:31] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=40MB]
[12:52:31] ^-- default region [type=default, persistence=false, lazyAlloc=true,
[12:52:31] ... initCfg=256MB, maxCfg=4898MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
[12:52:31] ^-- volatileDsMemPlc region [type=user, persistence=false, lazyAlloc=true,
[12:52:31] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
[12:52:32] Security status [authentication=off, sandbox=off, tls/ssl=off]
[12:52:32] Performance suggestions for grid (fix if possible)
[12:52:32] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[12:52:32] ^-- Specify JVM heap max size (add '-Xmx[g|G|m|M|k|K]' to JVM options)
[12:52:32] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options)
[12:52:32] Refer to this page for more performance suggestions: https://ignite.apache.org/docs/latest/perf-and-troubleshooting/memory-tuning
[12:52:32]
[12:52:32]
[12:52:32] Ignite node started OK (id=0ddda712)
[12:52:32] Topology snapshot [ver=3, locNode=0ddda712, servers=3, clients=0, state=ACTIVE, CPUs=8, offheap=9.6GB, heap=12.0GB]
[12:52:32] ^-- Baseline [id=0, size=3, online=3, offline=0]
Services deployed on node3: [myService8, myService2, myService7, myService4]
Now sleeping for 20s.
[12:52:42] Topology snapshot [ver=4, locNode=0ddda712, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=9.6GB, heap=12.0GB]
[12:52:42] Coordinator changed [prev=TcpDiscoveryNode [id=a6e3015d-6abd-4bf5-b352-bd291372cae4, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47500, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500, /192.168.161.44:47500], discPort=47500, order=1, intOrder=1, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false], cur=TcpDiscoveryNode [id=00dc7f4a-d0e4-4391-bd04-28b9ef934050, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/192.168.161.44:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false]]
[12:52:42] ^-- Baseline [id=0, size=2, online=2, offline=0]
[12:52:45] Topology snapshot [ver=5, locNode=0ddda712, servers=1, clients=0, state=ACTIVE, CPUs=8, offheap=4.8GB, heap=6.0GB]
[12:52:45] Coordinator changed [prev=TcpDiscoveryNode [id=00dc7f4a-d0e4-4391-bd04-28b9ef934050, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/192.168.161.44:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false], cur=TcpDiscoveryNode [id=0ddda712-bb41-4814-9124-86e010ce60c9, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47502, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/0:0:0:0:0:0:0:1:47502, /127.0.0.1:47502, /192.168.161.44:47502], discPort=47502, order=3, intOrder=3, loc=true, ver=2.17.0#20250209-sha1:d53d4540, isClient=false]]
[12:52:45] ^-- Baseline [id=0, size=1, online=1, offline=0]
Sep 03, 2025 12:52:45 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send message to remote node [node=TcpDiscoveryNode [id=00dc7f4a-d0e4-4391-bd04-28b9ef934050, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/192.168.161.44:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_EXCHANGE, topicOrd=31, ordered=false, timeout=0, skipOnTimeout=false, msg=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.latch.LatchAckMessage@63e33ef7]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): TcpDiscoveryNode [id=00dc7f4a-d0e4-4391-bd04-28b9ef934050, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/192.168.161.44:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false]
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:416)
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:692)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1181)
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:690)
at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:442)
at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:231)
at org.apache.ignite.spi.communication.tcp.internal.CommunicationWorker.processDisconnect(CommunicationWorker.java:376)
at org.apache.ignite.spi.communication.tcp.internal.CommunicationWorker.body(CommunicationWorker.java:174)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$3.body(TcpCommunicationSpi.java:848)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Sep 03, 2025 12:52:45 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send message to remote node [node=TcpDiscoveryNode [id=00dc7f4a-d0e4-4391-bd04-28b9ef934050, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/192.168.161.44:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage [parts=HashMap {-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=3, minorTopVer=2], updateSeq=110, size=100], -1365813811=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=3, minorTopVer=2], updateSeq=4, size=713]}, partCntrs=HashMap {-2100569601=CachePartitionPartialCountersMap {}, -1365813811=CachePartitionPartialCountersMap {656=(0,2)}}, partsSizes=HashMap {-1365813811=HashMap {656=1}}, partHistCntrs=null, err=null, client=false, exchangeStartTime=1756884163004, finishMsg=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=a6e3015d-6abd-4bf5-b352-bd291372cae4, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47500, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500, /192.168.161.44:47500], discPort=47500, order=1, intOrder=1, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false], topVer=4, msgTemplate=null, span=org.apache.ignite.internal.processors.tracing.NoopSpan@b7c450d, nodeId8=0ddda712, msg=Node left, type=NODE_LEFT, tstamp=1756884162968], nodeId=a6e3015d, evt=NODE_LEFT], lastVer=GridCacheVersion [topVer=368364147, order=1756884147363, nodeOrder=3, dataCenterId=0], super=GridCacheMessage [msgId=12, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): TcpDiscoveryNode [id=00dc7f4a-d0e4-4391-bd04-28b9ef934050, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.161.44:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.161.44], sockAddrs=HashSet [/192.168.161.44:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, loc=false, ver=2.17.0#20250209-sha1:d53d4540, isClient=false]
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:416)
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:692)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1181)
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:690)
at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:442)
at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:231)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1105)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1052)
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2059)
at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2152)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1209)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:2166)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendPartitions(GridDhtPartitionsExchangeFuture.java:2302)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1785)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:1055)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3321)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3155)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.base/java.lang.Thread.run(Thread.java:829)

Services deployed on node3: [myService8, myService1, myService2, myService7, myService4] (5 total)
[12:53:32]
[12:53:32] ^-- sysMemPlc region [type=internal, persistence=false, lazyAlloc=false,
[12:53:32] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.21%, allocRam=40MB]
[12:53:32] ^-- default region [type=default, persistence=false, lazyAlloc=true,
[12:53:32] ... initCfg=256MB, maxCfg=4898MB, usedRam=8MB, freeRam=99.84%, allocRam=256MB]
[12:53:32] ^-- volatileDsMemPlc region [type=user, persistence=false, lazyAlloc=true,
[12:53:32] ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
[12:53:32]
[12:53:32]
[12:53:32] Data storage metrics for local node (to disable set 'metricsLogFrequency' to 0)
[12:53:32] ^-- Off-heap memory [used=8MB, free=99.83%, allocated=296MB]
[12:53:32] ^-- Page memory [pages=2250]
[12:53:42] Ignite node stopped OK [uptime=00:01:09.829]

Process finished with exit code 130
```

Reproduced with Ignite 2.17.0, OpenLogic OpenJDK 11.0.22.

Start by running Node1And2.java and Node3.java together to reproduce the topology change and compare the service descriptor output before and after the two nodes leave. Trace the service redeployment path exercised by this scenario, using ServiceImpl.java as the minimal service implementation. Done means all 10 services are redeployed on the remaining node after both nodes stop.

Written by the indexing model from the issue text.