apache / apache/ignite

Ignite client nodes are getting terminated when we stop the ignite server.

Open
#11,680 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

ThickClient mode is in use. There is an issue where the client terminates when the ignite server is stopped.
The error occurs after the join timeout time has expired.
version : 2.16.0

Even if the server is stopped, can't the client continue to wait without being terminated?
Similar symptoms were seen in the previous issue
[https://issues.apache.org/jira/browse/IGNITE-20690](url)

[client node configuration]
```igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setClientMode(true);
igniteConfiguration.setIgniteInstanceName("server_instance");

TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
tcpDiscoverySpi.setSocketTimeout(10000)
.setJoinTimeout(60000)
.setNetworkTimeout(10000);
tcpDiscoverySpi.setMaxAckTimeout(10000);
tcpDiscoverySpi.setReconnectCount(1);

KubernetesConnectionConfiguration kcfg = new KubernetesConnectionConfiguration();
kcfg.setNamespace(namespace);
kcfg.setServiceName(serviceName);

TcpDiscoveryKubernetesIpFinder finder = new TcpDiscoveryKubernetesIpFinder(kcfg);
tcpDiscoverySpi.setIpFinder(finder);

igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);

TcpCommunicationSpi tcpCommunicationSpi = new TcpCommunicationSpi();
tcpCommunicationSpi.setMessageQueueLimit(1024);
igniteConfiguration.setCommunicationSpi(tcpCommunicationSpi);

CacheConfiguration cacheConfiguration = new CacheConfiguration("IgniteCache");
cacheConfiguration.setDataRegionName("DataStorageRegion");
cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setBackups(1);
cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
cacheConfiguration.setPartitionLossPolicy(PartitionLossPolicy.READ_ONLY_SAFE);
cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);
cacheConfiguration.setOnheapCacheEnabled(false);
igniteConfiguration.setCacheConfiguration(cacheConfiguration);

[error]
[2024-11-25 13:47:15.922] [WARN ] [disco-event-worker-#49%server_instance%] Local node SEGMENTED: TcpDiscoveryNode [id=62635b50-c078-42f8-b710-02c6f08462c3, consistentId=fb457a44-d27f-418e-8583-a51f9a26b518, addrs=ArrayList [127.0.0.1, 172.17.0.1, 172.18.0.1, 172.19.0.1, 172.31.207.9], sockAddrs=HashSet [/172.17.0.1:0, /127.0.0.1:0, /172.19.0.1:0, /172.18.0.1:0, /172.31.207.9:0], discPort=0, order=2, intOrder=0, lastExchangeTime=1732509929452, loc=true, ver=2.16.0#20231215-sha1:7bde6a42, isClient=true]
[2024-11-25 13:47:15.925] [ERROR] [disco-event-worker-#49%server_instance%] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SEGMENTATION, err=null]]
[2024-11-25 13:47:15.926] [ERROR] [disco-event-worker-#49%server_instance%] No deadlocked threads detected.
[2024-11-25 13:47:15.961] [ERROR] [disco-event-worker-#49%server_instance%] Thread dump at 2024/11/25 13:47:15 KST```

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the provided Ignite 2.16.0 ThickClient configuration, Kubernetes discovery settings, and join timeout values. Start by tracing the discovery behavior that emits the SEGMENTED and StopNodeFailureHandler messages after the server stops, and compare it with IGNITE-20690. Done means the client does not terminate solely because the server is stopped and the intended waiting or reconnection behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.