Ignite client nodes are getting terminated when we stop the ignite server.
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.9k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
ThickClient mode is in use. There is an issue where the client terminates when the ignite server is stopped.
The error occurs after the join timeout time has expired.
version : 2.16.0
Even if the server is stopped, can't the client continue to wait without being terminated?
Similar symptoms were seen in the previous issue
[https://issues.apache.org/jira/browse/IGNITE-20690](url)
[client node configuration]
```igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setClientMode(true);
igniteConfiguration.setIgniteInstanceName("server_instance");
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
tcpDiscoverySpi.setSocketTimeout(10000)
.setJoinTimeout(60000)
.setNetworkTimeout(10000);
tcpDiscoverySpi.setMaxAckTimeout(10000);
tcpDiscoverySpi.setReconnectCount(1);
KubernetesConnectionConfiguration kcfg = new KubernetesConnectionConfiguration();
kcfg.setNamespace(namespace);
kcfg.setServiceName(serviceName);
TcpDiscoveryKubernetesIpFinder finder = new TcpDiscoveryKubernetesIpFinder(kcfg);
tcpDiscoverySpi.setIpFinder(finder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
TcpCommunicationSpi tcpCommunicationSpi = new TcpCommunicationSpi();
tcpCommunicationSpi.setMessageQueueLimit(1024);
igniteConfiguration.setCommunicationSpi(tcpCommunicationSpi);
CacheConfiguration cacheConfiguration = new CacheConfiguration("IgniteCache");
cacheConfiguration.setDataRegionName("DataStorageRegion");
cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setBackups(1);
cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
cacheConfiguration.setPartitionLossPolicy(PartitionLossPolicy.READ_ONLY_SAFE);
cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);
cacheConfiguration.setOnheapCacheEnabled(false);
igniteConfiguration.setCacheConfiguration(cacheConfiguration);
[error]
[2024-11-25 13:47:15.922] [WARN ] [disco-event-worker-#49%server_instance%] Local node SEGMENTED: TcpDiscoveryNode [id=62635b50-c078-42f8-b710-02c6f08462c3, consistentId=fb457a44-d27f-418e-8583-a51f9a26b518, addrs=ArrayList [127.0.0.1, 172.17.0.1, 172.18.0.1, 172.19.0.1, 172.31.207.9], sockAddrs=HashSet [/172.17.0.1:0, /127.0.0.1:0, /172.19.0.1:0, /172.18.0.1:0, /172.31.207.9:0], discPort=0, order=2, intOrder=0, lastExchangeTime=1732509929452, loc=true, ver=2.16.0#20231215-sha1:7bde6a42, isClient=true]
[2024-11-25 13:47:15.925] [ERROR] [disco-event-worker-#49%server_instance%] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SEGMENTATION, err=null]]
[2024-11-25 13:47:15.926] [ERROR] [disco-event-worker-#49%server_instance%] No deadlocked threads detected.
[2024-11-25 13:47:15.961] [ERROR] [disco-event-worker-#49%server_instance%] Thread dump at 2024/11/25 13:47:15 KST```
Contributor guide
Research direction
Reproduce the issue with the provided Ignite 2.16.0 ThickClient configuration, Kubernetes discovery settings, and join timeout values. Start by tracing the discovery behavior that emits the SEGMENTED and StopNodeFailureHandler messages after the server stops, and compare it with IGNITE-20690. Done means the client does not terminate solely because the server is stopped and the intended waiting or reconnection behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100