finos / finos/symphony-bdk-java
Datafeed Retry Fails if Agent is down with LB in Front
- Dominant language
- Java
- Stars
- 29
- Forks
- 77
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 3
Description
## Bug Report
### Steps to Reproduce:
1. Setup an LB with an Agent.
2. Start the Agent.
3. Connect BDK to Agent.
4. Stop the Agent.
5. See that it stops trying to recover.
### Expected Result:
It should continue to retry as this scenario can happen if maintenance is being done on the agent behind the LB or if even only one of the Agents becomes unhealthy and it happens to reach the agent before LB can take it off rotation.
### Actual Result:
```
2025-07-01T10:44:36.332-04:00 ERROR 29488 --- [onyBdk_Datafeed] c.s.b.c.s.d.impl.AbstractDatafeedLoop : Network error occurred while trying to connect to the "AGENT" at the following address: https://agent.symphony.com:443/agent. Error while trying to validate certificate for the trust store. This type of error typically means that your network is using a self-signed certificate.
{}
jakarta.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:534) ~[jersey-apache-connector-3.1.9.jar:na]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:674) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:709) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:703) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:391) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:703) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:673) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:439) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:345) ~[jersey-client-3.1.9.jar:na]
at com.symphony.bdk.http.jersey2.ApiClientJersey2.getResponse(ApiClientJersey2.java:207) ~[symphony-bdk-http-jersey2-3.1.1.jar:3.1.1]
at com.symphony.bdk.http.jersey2.ApiClientJersey2.invokeAPI(ApiClientJersey2.java:158) ~[symphony-bdk-http-jersey2-3.1.1.jar:3.1.1]
at com.symphony.bdk.gen.api.DatafeedApi.readDatafeedWithHttpInfo(DatafeedApi.java:388) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.gen.api.DatafeedApi.readDatafeed(DatafeedApi.java:321) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.DatafeedLoopV2.readEvents(DatafeedLoopV2.java:134) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.AbstractAckIdEventLoop.readAndHandleEvents(AbstractAckIdEventLoop.java:42) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.retry.RetryWithRecovery.executeOnce(RetryWithRecovery.java:105) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery.lambda$execute$1(Resilience4jRetryWithRecovery.java:65) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at io.github.resilience4j.retry.Retry.lambda$decorateCheckedSupplier$1(Retry.java:135) ~[resilience4j-retry-2.2.0.jar:2.2.0]
at io.github.resilience4j.retry.Retry.executeCheckedSupplier(Retry.java:350) ~[resilience4j-retry-2.2.0.jar:2.2.0]
at com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery.execute(Resilience4jRetryWithRecovery.java:65) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.DatafeedLoopV2.runLoop(DatafeedLoopV2.java:103) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.AbstractDatafeedLoop.start(AbstractDatafeedLoop.java:90) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.AbstractAckIdEventLoop.start(AbstractAckIdEventLoop.java:19) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.spring.service.LoopAsyncLauncherService.uncheckedStart(LoopAsyncLauncherService.java:86) ~[symphony-bdk-core-spring-boot-starter-3.1.1.jar:3.1.1]
at com.symphony.bdk.http.api.tracing.MDCUtils$MdcRunnable.run(MDCUtils.java:59) ~[symphony-bdk-http-api-3.1.1.jar:3.1.1]
at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1719) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1518) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[na:na]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.14.jar:4.5.14]
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:486) ~[jersey-apache-connector-3.1.9.jar:na]
... 29 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) ~[na:na]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510) ~[na:na]
... 44 common frames omitted
2025-07-01T10:44:36.338-04:00 INFO 29488 --- [ionShutdownHook] c.s.b.c.s.d.impl.AbstractDatafeedLoop : Stopping the datafeed loop (will happen once the current read is finished)...
2025-07-01T10:44:36.339-04:00 INFO 29488 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2025-07-01T10:44:36.341-04:00 INFO 29488 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2025-07-01T10:44:36.346-04:00 INFO 29488 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
```
### Environment:
BDK 3.1.1 w/ Agent behind an LB.
Contributor guide
Research direction
Start with DatafeedLoopV2.runLoop and AbstractDatafeedLoop.start, then trace RetryWithRecovery.executeOnce and Resilience4jRetryWithRecovery.execute in the stack trace. Reproduce the load-balancer scenario by stopping the Agent and observe the retry behavior. Done means the datafeed continues trying to recover after the Agent becomes unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100