Hazelcast eureka client is not able to register after Eureka Server restart

Hi,

Currently we use Eureka Server with Spring Boot 2.5.5 - org.springframework.cloud:spring-cloud-starter-netflix-eureka-server:jar:3.0.4 and we plan to upgrade it to Spring Boot 3.1.6 - org.springframework.cloud:spring-cloud-starter-netflix-eureka-server:jar:4.0.3.

Unfortunately, there is an issue with newer version of Eureka Server. After restarting Eureka Server cache's eureka client is not able to register itself but our application's eureka client is able to do that. Looks like an issue related with cache eureka client.

In the logs of eureka server we can see some warnings:

>2024-01-22T15:45:48.487+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.s.a.dao.DaoAuthenticationProvider : Authenticated user
2024-01-22T15:45:48.487+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.s.w.a.www.BasicAuthenticationFilter : Set SecurityContextHolder to UsernamePasswordAuthenticationToken [Principal=org.springframework.security.core.userdetails.User [Username=admin, Password=[PROTECTED], Enabled=true, AccountNonExpired=true, credentialsNonExpired=true, AccountNonLocked=true, Granted Authorities=[ROLE_ADMIN, ROLE_USER]], Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=127.0.0.1, SessionId=null], Granted Authorities=[ROLE_ADMIN, ROLE_USER]]
2024-01-22T15:45:48.488+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.security.web.FilterChainProxy : Secured PUT /eureka/apps/USER-CACHE/f0899c53-47a8-4a2b-bab4-99855fe3c55c?status=UP&lastDirtyTimestamp=1705934237735
2024-01-22T15:45:48.489+01:00 WARN 33229 --- [nio-8761-exec-9] c.n.e.registry.AbstractInstanceRegistry : DS: Registry: lease doesn't exist, registering resource: USER-CACHE - f0899c53-47a8-4a2b-bab4-99855fe3c55c
2024-01-22T15:45:48.489+01:00 WARN 33229 --- [nio-8761-exec-9] c.n.eureka.resources.InstanceResource : Not Found (Renew): USER-CACHE - f0899c53-47a8-4a2b-bab4-99855fe3c55c
2024-01-22T15:45:48.489+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.security.web.FilterChainProxy : Securing PUT /error?status=UP&lastDirtyTimestamp=1705934237735
2024-01-22T15:45:48.490+01:00 DEBUG 33229 --- [nio-8761-exec-9] o.s.security.web.FilterChainProxy : Secured PUT /error?status=UP&lastDirtyTimestamp=1705934237735

In the logs of our application we have an error related to cache:

>2024-01-22T15:51:48.563+01:00 ERROR 32183 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_USER-CACHE/f0899c53-47a8-4a2b-bab4-99855fe3c55c - was unable to send heartbeat!
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:112)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:837)
at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1401)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

Bellow our cache configuration:

```
@Bean
public CacheManager cacheManager(HazelcastInstance hazelcastInstance) {
log.debug("Starting HazelcastCacheManager");
return new com.hazelcast.spring.cache.HazelcastCacheManager(hazelcastInstance);
}

@Bean
public HazelcastInstance hazelcastInstance() {
log.debug("Configuring Hazelcast");
HazelcastInstance hazelCastInstance = Hazelcast.getHazelcastInstanceByName(appName);
if (hazelCastInstance != null) {
log.debug("Hazelcast already initialized");
return hazelCastInstance;
}
Config config = new Config();
config.setInstanceName(appName);
config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
config
.getNetworkConfig()
.getJoin()
.getEurekaConfig()
.setEnabled(true)
.setProperty("self-registration", "true")
.setProperty("namespace", "hazelcast")
.setProperty("use-metadata-for-host-and-port", "false")
.setProperty("skip-eureka-registration-verification", "true")
.setProperty("use-classpath-eureka-client-props", "false")
.setProperty("shouldUseDns", "false")
.setProperty("name", appName + "-cache")
.setProperty("serviceUrl.default", eurekaServiceUrl);
config.setManagementCenterConfig(new ManagementCenterConfig());
// config.addMapConfig(initializeDefaultMapConfig(jHipsterProperties));
// config.addMapConfig(initializeDomainMapConfig(jHipsterProperties));
return Hazelcast.newHazelcastInstance(config);
}
```

Start by tracing the Hazelcast Eureka configuration shown in the report and the DiscoveryClient heartbeat and renew path named in the stack trace. Reproduce a Eureka Server restart with the provided setup, then verify that the cache client can register and resume heartbeats afterward.

Written by the indexing model from the issue text.