Add retry logic in ExampleUtils that handles network flakes.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
This exception is thrown when there is a network flake when running the traffic examples. Should we retry instead of throwing the exception and failing the job?
`Exception in thread "main" java.io.IOException: Error getting access token for service account: `
` at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:319)`
` at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149)`
` at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:135)`
` at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)`
` at com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize(ChainingHttpRequestInitializer.java:52)`
` at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)`
` at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)`
` at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)`
` at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)`
` at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)`
` at org.apache.beam.examples.common.ExampleUtils.executeNullIfNotFound(ExampleUtils.java:397)`
` at org.apache.beam.examples.common.ExampleUtils.setupPubsubTopic(ExampleUtils.java:283)`
` at org.apache.beam.examples.common.ExampleUtils.setupPubsub(ExampleUtils.java:131)`
` at org.apache.beam.examples.common.ExampleUtils.setup(ExampleUtils.java:105)`
` at org.apache.beam.examples.complete.TrafficMaxLaneFlow.main(TrafficMaxLaneFlow.java:334)`
`Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake`
` at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)`
` at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1379)`
` at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1407)`
` at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)`
` at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)`
` at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)`
` at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)`
` at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)`
` at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)`
` at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)`
` at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)`
` at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:317)`
` ... 14 more`
`Caused by: java.io.EOFException: SSL peer shut down incorrectly`
` at sun.security.ssl.InputRecord.read(InputRecord.java:505)`
` at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)`
` ... 25 more`
Add retry code somewhere here?
https://github.com/apache/beam/blob/29859eb54d05b96a9db477e7bb04537510273bd2/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java#L398
Imported from Jira [BEAM-3764](https://issues.apache.org/jira/browse/BEAM-3764). Original Jira may contain additional context.
Reported by: davidyan.
Contributor guide
Research direction
Start in examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java around executeNullIfNotFound at line 398, then trace the setupPubsub and setup entry points shown in the stack trace. Determine how the reported SSL handshake and EOF failures should be distinguished from permanent errors; done means transient network flakes are retried while genuine failures still surface.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100