Azure / Azure/azure-libraries-for-java

Timeout exceptions with various APIs

Open
#799 16 comments 0 reactions 1 assignee Assigned to @weidongxu-microsoft View on GitHub
ARM customer-reported customer-response-expected Mgmt
Dominant language
Java
Stars
97
Forks
102
PR merge metrics
No merged PRs in 30d

Description

**Query/Question**
We are seeing more timeout exceptions calling various Azure APIs these days. All of them failed with socket closed then timeout exception. What can be the cause of these issues? Will increase read timeout in Azure client help?

The two most common API calls that failed are Azure.virtualMachines().getByResourceGroup() and Azure.managementLocks().deleteByIds(). We are using 1.12.0 SDK. Here is one example stack trace,

```
java.lang.RuntimeException: java.net.SocketTimeoutException: timeout
at rx.exceptions.Exceptions.propagate(Exceptions.java:58) ~[?:?]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:464) ~[?:?]
at rx.observables.BlockingObservable.last(BlockingObservable.java:227) ~[?:?]
at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:110) ~[?:?]
at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:34) ~[?:?]
at com.adobe.ebs.backup.ec2.api.AzureVMApi$1.call(AzureVMApi.java:106) ~[?:?]
...
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.net.SocketTimeoutException: timeout
at okio.Okio$3.newTimeoutException(Okio.java:210) ~[?:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[?:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[?:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[?:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[?:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[?:?]
at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) ~[?:?]
at okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) ~[?:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.LoggingInterceptor.intercept(LoggingInterceptor.java:116) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:51) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) ~[?:?]
at okhttp3.RealCall.execute(RealCall.java:60) ~[?:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:171) ~[?:?]
at rx.Subscriber.setProducer(Subscriber.java:211) ~[?:?]
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153) ~[?:?]
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32) ~[?:?]
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
at rx.Observable.subscribe(Observable.java:10238) ~[?:?]
at rx.Observable.subscribe(Observable.java:10205) ~[?:?]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[?:?]
... 23 more
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:?]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[?:?]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) ~[?:?]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[?:?]
at okio.Okio$2.read(Okio.java:138) ~[?:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[?:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[?:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[?:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[?:?]
at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) ~[?:?]
at okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) ~[?:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.LoggingInterceptor.intercept(LoggingInterceptor.java:116) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:51) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[?:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[?:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) ~[?:?]
at okhttp3.RealCall.execute(RealCall.java:60) ~[?:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:171) ~[?:?]
at rx.Subscriber.setProducer(Subscriber.java:211) ~[?:?]
at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152) ~[?:?]
at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[?:?]
at rx.Observable.unsafeSubscribe(Observable.java:10142) ~[?:?]
at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153) ~[?:?]
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32) ~[?:?]
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[?:?]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[?:?]
at rx.Observable.subscribe(Observable.java:10238) ~[?:?]
at rx.Observable.subscribe(Observable.java:10205) ~[?:?]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444) ~[?:?]
... 23 more
```

**Setup (please complete the following information if applicable):**
- OS: CentOS 7
- IDE : Eclipse
- Version of the Library used: 1.12.0

**Information Checklist**
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report
- [x] Query Added
- [x] Setup information Added

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.