Azure / Azure/azure-libraries-for-java

Timeout exceptions with various APIs

Open
#799 16 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

ARM customer-reported customer-response-expected Mgmt
Dominant language
Java
Stars
97
Forks
102
No merged PRs in 30d

Description

Query/Question
We are seeing more timeout exceptions calling various Azure APIs these days. All of them failed with socket closed then timeout exception. What can be the cause of these issues? Will increase read timeout in Azure client help?

The two most common API calls that failed are Azure.virtualMachines().getByResourceGroup() and Azure.managementLocks().deleteByIds(). We are using 1.12.0 SDK. Here is one example stack trace,

java.lang.RuntimeException: java.net.SocketTimeoutException: timeout
	at rx.exceptions.Exceptions.propagate(Exceptions.java:58) ~[?:?]
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:464) ~[?:?]
	at rx.observables.BlockingObservable.last(BlockingObservable.java:227) ~[?:?]
	at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:110) ~[?:?]
	at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:34) ~[?:?]
	at com.adobe.ebs.backup.ec2.api.AzureVMApi$1.call(AzureVMApi.java:106) ~[?:?]
...
	at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.net.SocketTimeoutException: timeout
	at okio.Okio$3.newTimeoutException(Okio.java:210) ~[?:?]
	at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[?:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[?:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[?:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[?:?]
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[?:?]
	at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) ~[?:?]
	at okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) ~[?:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.LoggingInterceptor.intercept(LoggingInterceptor.java:116) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:51) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) ~[?:?]
	at okhttp3.RealCall.execute(RealCall.java:60) ~[?:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[?:?]
	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:171) ~[?:?]
	at rx.Subscriber.setProducer(Subscriber.java:211) ~[?:?]
	at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) ~[?:?]
	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152) ~[?:?]
	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153) ~[?:?]
	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32) ~[?:?]
	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
	at rx.Observable.subscribe(Observable.java:10238) ~[?:?]
	at rx.Observable.subscribe(Observable.java:10205) ~[?:?]
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[?:?]
	... 23 more
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:?]
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:?]
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[?:?]
	at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) ~[?:?]
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[?:?]
	at okio.Okio$2.read(Okio.java:138) ~[?:?]
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[?:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[?:?]
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[?:?]
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[?:?]
	at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) ~[?:?]
	at okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) ~[?:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.LoggingInterceptor.intercept(LoggingInterceptor.java:116) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:51) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) ~[?:?]
	at okhttp3.RealCall.execute(RealCall.java:60) ~[?:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[?:?]
	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:171) ~[?:?]
	at rx.Subscriber.setProducer(Subscriber.java:211) ~[?:?]
	at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) ~[?:?]
	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152) ~[?:?]
	at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
	at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
	at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153) ~[?:?]
	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32) ~[?:?]
	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
	at rx.Observable.subscribe(Observable.java:10238) ~[?:?]
	at rx.Observable.subscribe(Observable.java:10205) ~[?:?]
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[?:?]
	... 23 more

Setup (please complete the following information if applicable):

  • OS: CentOS 7
  • IDE : Eclipse
  • Version of the Library used: 1.12.0

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • Query Added
  • Setup information Added

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.