apache / apache/beam

Add retry logic in ExampleUtils that handles network flakes.

Open
#18,624 0 comments 0 reactions 0 assignees View on GitHub
examples improvement java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

This exception is thrown when there is a network flake when running the traffic examples. Should we retry instead of throwing the exception and failing the job?

`Exception in thread "main" java.io.IOException: Error getting access token for service account: `
`        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:319)`
`        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149)`
`        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:135)`
`        at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)`
`        at com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize(ChainingHttpRequestInitializer.java:52)`
`        at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)`
`        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)`
`        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)`
`        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)`
`        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)`
`        at org.apache.beam.examples.common.ExampleUtils.executeNullIfNotFound(ExampleUtils.java:397)`
`        at org.apache.beam.examples.common.ExampleUtils.setupPubsubTopic(ExampleUtils.java:283)`
`        at org.apache.beam.examples.common.ExampleUtils.setupPubsub(ExampleUtils.java:131)`
`        at org.apache.beam.examples.common.ExampleUtils.setup(ExampleUtils.java:105)`
`        at org.apache.beam.examples.complete.TrafficMaxLaneFlow.main(TrafficMaxLaneFlow.java:334)`
`Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake`
`        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)`
`        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1379)`
`        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1407)`
`        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)`
`        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)`
`        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)`
`        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)`
`        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)`
`        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)`
`        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)`
`        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)`
`        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:317)`
`        ... 14 more`
`Caused by: java.io.EOFException: SSL peer shut down incorrectly`
`        at sun.security.ssl.InputRecord.read(InputRecord.java:505)`
`        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)`
`        ... 25 more`

Add retry code somewhere here?

https://github.com/apache/beam/blob/29859eb54d05b96a9db477e7bb04537510273bd2/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java#L398

Imported from Jira [BEAM-3764](https://issues.apache.org/jira/browse/BEAM-3764). Original Jira may contain additional context.
Reported by: davidyan.

Contributor guide

Open the contributing guide

Research direction

Start in examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java around executeNullIfNotFound at line 398, then trace the setupPubsub and setup entry points shown in the stack trace. Determine how the reported SSL handshake and EOF failures should be distinguished from permanent errors; done means transient network flakes are retried while genuine failures still surface.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.