Azure / Azure/azure-sdk-for-java
[BUG] Deadlock when using `DefaultAzureCredential`s and virtual threads are available
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 2.2k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 178
Description
**Describe the bug**
When using two or more `DefaultAzureCredential` instances to automatically discover Service Principal credentials from the environment _and_ virtual threads are available and used by the `SharedExecutorService`, making any request (in our case blob storage listing) can lead to a deadlock in retrieving tokens. This is because acquiring a token is **pinning the virtual thread**. With only a few carrier threads available, this is quickly leading to a deadlock where the Azure SDK, and virtual threads in general, become unusable.
***Exception or Stack Trace***
First carrier thread
"ForkJoinPool-1-worker-10" daemon prio=5 tid=416 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.lang.VirtualThread.parkOnCarrierThread()
local variable: java.lang.VirtualThread#2
local variable: jdk.internal.event.VirtualThreadPinnedEvent#1
at java.lang.VirtualThread.park()
local variable: java.lang.VirtualThread#2
at java.lang.System$2.parkVirtualThread()
local variable: java.lang.System$2#1
local variable: java.lang.VirtualThread#2
local variable: java.lang.VirtualThread#2
at jdk.internal.misc.VirtualThreads.park()
at java.util.concurrent.locks.LockSupport.park()
local variable: java.lang.VirtualThread#2
at java.util.concurrent.CompletableFuture$Signaller.block()
local variable: java.util.concurrent.CompletableFuture$Signaller#1
at java.util.concurrent.ForkJoinPool.unmanagedBlock()
local variable: java.util.concurrent.CompletableFuture$Signaller#1
at java.util.concurrent.ForkJoinPool.managedBlock()
at java.util.concurrent.CompletableFuture.waitingGet()
local variable: java.util.concurrent.CompletableFuture#19
local variable: java.util.concurrent.CompletableFuture$Signaller#1
at java.util.concurrent.CompletableFuture.get()
local variable: java.util.concurrent.CompletableFuture#19
at com.azure.identity.implementation.IdentitySyncClient.authenticateWithConfidentialClientCache(IdentitySyncClient.java:173)
local variable: com.azure.identity.implementation.IdentitySyncClient#9
local variable: com.azure.core.credential.TokenRequestContext#3
local variable: com.microsoft.aad.msal4j.ConfidentialClientApplication#1
local variable: com.microsoft.aad.msal4j.SilentParameters$SilentParametersBuilder#1
at com.azure.identity.ClientSecretCredential.getTokenSync(ClientSecretCredential.java:124)
local variable: com.azure.identity.ClientSecretCredential#3
local variable: com.azure.core.credential.TokenRequestContext#3
at com.azure.identity.EnvironmentCredential.getTokenSync(EnvironmentCredential.java:186)
local variable: com.azure.identity.EnvironmentCredential#3
local variable: com.azure.core.credential.TokenRequestContext#3
at com.azure.identity.ChainedTokenCredential.getTokenSync(ChainedTokenCredential.java:130)
local variable: com.azure.identity.DefaultAzureCredential#3
local variable: com.azure.core.credential.TokenRequestContext#3
local variable: java.util.ArrayList#6042
at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
local variable: com.azure.core.implementation.AccessTokenCache#3
local variable: com.azure.identity.DefaultAzureCredential#3
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#3
at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
local variable: com.azure.core.implementation.AccessTokenCache#3
local variable: com.azure.core.credential.TokenRequestContext#1
local variable: com.azure.core.implementation.AccessTokenCacheInfo#3
local variable: com.azure.identity.implementation.MsalToken#2
local variable: java.time.OffsetDateTime#4
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#3
local variable: com.azure.identity.implementation.MsalToken#2
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40#1
at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
local variable: com.azure.core.implementation.AccessTokenCache#3
local variable: com.azure.core.credential.TokenRequestContext#1
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.credential.TokenRequestContext#1
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.credential.TokenRequestContext#1
at com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy.authorizeRequestSync(StorageBearerTokenChallengeAuthorizationPolicy.java:63)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: java.lang.String[]#1260
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.processSync(BearerTokenAuthenticationPolicy.java:174)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#2
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
at com.azure.storage.common.policy.MetadataValidationPolicy.processSync(MetadataValidationPolicy.java:42)
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#3
at com.azure.core.http.policy.AddHeadersFromContextPolicy.processSync(AddHeadersFromContextPolicy.java:67)
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#3
at com.azure.core.http.policy.AddDatePolicy.processSync(AddDatePolicy.java:50)
local variable: com.azure.core.http.policy.AddDatePolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.core.http.policy.AddDatePolicy#3
at com.azure.storage.common.policy.RequestRetryPolicy.lambda$attemptSync$2(RequestRetryPolicy.java:261)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0.call()
local variable: com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0#1
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:62)
local variable: reactor.core.publisher.MonoCallable#15
local variable: java.time.Duration#229
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:55)
local variable: reactor.core.publisher.MonoCallable#15
at com.azure.storage.common.policy.RequestRetryPolicy.attemptSync(RequestRetryPolicy.java:270)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.HttpRequest#3
local variable: reactor.core.publisher.MonoCallable#15
at com.azure.storage.common.policy.RequestRetryPolicy.processSync(RequestRetryPolicy.java:71)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.HttpRequest#3
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.storage.common.policy.RequestRetryPolicy#3
at com.azure.core.http.policy.RequestIdPolicy.processSync(RequestIdPolicy.java:77)
local variable: com.azure.core.http.policy.RequestIdPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.policy.RequestIdPolicy#3
at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
local variable: com.azure.core.http.policy.UserAgentPolicy$1#4
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
local variable: com.azure.core.http.policy.UserAgentPolicy#4
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.policy.UserAgentPolicy#4
at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
local variable: com.azure.core.http.HttpPipeline#3
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#26
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#26
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
local variable: com.azure.core.util.tracing.Utils$$Lambda+0x000078380135f268#1
at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#26
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: java.lang.Object[]#9601
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
local variable: com.azure.core.http.rest.RestProxy#26
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.reflect.Method#8034
local variable: java.lang.Object[]#9601
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.util.Context$1#1
at jdk.proxy2.$Proxy213.listBlobFlatSegmentSync()
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.String#70505
local variable: java.lang.String#102455
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#39616
local variable: java.lang.String#102453
local variable: java.lang.String#139122
local variable: com.azure.core.util.Context$1#1
at com.azure.storage.blob.implementation.ContainersImpl.listBlobFlatSegmentWithResponse(ContainersImpl.java:5959)
local variable: com.azure.storage.blob.implementation.ContainersImpl#5
local variable: java.lang.String#102455
local variable: java.lang.String#39616
local variable: com.azure.core.util.Context$1#1
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#139122
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$6(BlobContainerClient.java:1046)
local variable: com.azure.storage.blob.BlobContainerClient#5
local variable: com.azure.storage.blob.models.ListBlobsOptions#1
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760.call()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#1
at com.azure.storage.common.implementation.StorageImplUtils.sendRequest(StorageImplUtils.java:494)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#1
local variable: class com.azure.storage.blob.models.BlobStorageException
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$7(BlobContainerClient.java:1050)
local variable: com.azure.storage.blob.BlobContainerClient#5
local variable: com.azure.storage.blob.models.ListBlobsOptions#2
local variable: com.azure.storage.blob.models.ListBlobsOptions#1
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#1
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#1
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$8(BlobContainerClient.java:1065)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#1
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#1
at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#1
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#1
at com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40.getPage()
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#1
at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#1
local variable: java.util.concurrent.atomic.AtomicBoolean#270
at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.(ContinuablePagedByItemIterable.java:83)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#1
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#1
local variable: com.azure.core.http.rest.PagedIterableBase$$Lambda+0x00007838015367b0#1
at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:58)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable#1
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#1
at java.lang.Iterable.spliterator()
at com.azure.core.util.paging.ContinuablePagedIterable.stream(ContinuablePagedIterable.java:85)
local variable: com.azure.core.http.rest.PagedIterable#1
Second carrier thread
"ForkJoinPool-1-worker-9" daemon prio=5 tid=417 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.lang.VirtualThread.parkOnCarrierThread()
local variable: java.lang.VirtualThread#3
local variable: jdk.internal.event.VirtualThreadPinnedEvent#2
at java.lang.VirtualThread.park()
local variable: java.lang.VirtualThread#3
at java.lang.System$2.parkVirtualThread()
local variable: java.lang.System$2#1
local variable: java.lang.VirtualThread#3
local variable: java.lang.VirtualThread#3
at jdk.internal.misc.VirtualThreads.park()
at java.util.concurrent.locks.LockSupport.park()
local variable: java.lang.VirtualThread#3
at java.util.concurrent.CompletableFuture$Signaller.block()
local variable: java.util.concurrent.CompletableFuture$Signaller#2
at java.util.concurrent.ForkJoinPool.unmanagedBlock()
local variable: java.util.concurrent.CompletableFuture$Signaller#2
at java.util.concurrent.ForkJoinPool.managedBlock()
at java.util.concurrent.CompletableFuture.waitingGet()
local variable: java.util.concurrent.CompletableFuture#21
local variable: java.util.concurrent.CompletableFuture$Signaller#2
at java.util.concurrent.CompletableFuture.get()
local variable: java.util.concurrent.CompletableFuture#21
at com.azure.identity.implementation.IdentitySyncClient.authenticateWithConfidentialClientCache(IdentitySyncClient.java:173)
local variable: com.azure.identity.implementation.IdentitySyncClient#18
local variable: com.azure.core.credential.TokenRequestContext#4
local variable: com.microsoft.aad.msal4j.ConfidentialClientApplication#2
local variable: com.microsoft.aad.msal4j.SilentParameters$SilentParametersBuilder#2
at com.azure.identity.ClientSecretCredential.getTokenSync(ClientSecretCredential.java:124)
local variable: com.azure.identity.ClientSecretCredential#6
local variable: com.azure.core.credential.TokenRequestContext#4
at com.azure.identity.EnvironmentCredential.getTokenSync(EnvironmentCredential.java:186)
local variable: com.azure.identity.EnvironmentCredential#6
local variable: com.azure.core.credential.TokenRequestContext#4
at com.azure.identity.ChainedTokenCredential.getTokenSync(ChainedTokenCredential.java:130)
local variable: com.azure.identity.DefaultAzureCredential#6
local variable: com.azure.core.credential.TokenRequestContext#4
local variable: java.util.ArrayList#6043
at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
local variable: com.azure.core.implementation.AccessTokenCache#6
local variable: com.azure.identity.DefaultAzureCredential#6
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#6
at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
local variable: com.azure.core.implementation.AccessTokenCache#6
local variable: com.azure.core.credential.TokenRequestContext#2
local variable: com.azure.core.implementation.AccessTokenCacheInfo#4
local variable: com.azure.identity.implementation.MsalToken#1
local variable: java.time.OffsetDateTime#5
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#6
local variable: com.azure.identity.implementation.MsalToken#1
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40#2
at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
local variable: com.azure.core.implementation.AccessTokenCache#6
local variable: com.azure.core.credential.TokenRequestContext#2
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.credential.TokenRequestContext#2
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.credential.TokenRequestContext#2
at com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy.authorizeRequestSync(StorageBearerTokenChallengeAuthorizationPolicy.java:63)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: java.lang.String[]#1261
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.processSync(BearerTokenAuthenticationPolicy.java:174)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#5
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
at com.azure.storage.common.policy.MetadataValidationPolicy.processSync(MetadataValidationPolicy.java:42)
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#6
at com.azure.core.http.policy.AddHeadersFromContextPolicy.processSync(AddHeadersFromContextPolicy.java:67)
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#6
at com.azure.core.http.policy.AddDatePolicy.processSync(AddDatePolicy.java:50)
local variable: com.azure.core.http.policy.AddDatePolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.core.http.policy.AddDatePolicy#6
at com.azure.storage.common.policy.RequestRetryPolicy.lambda$attemptSync$2(RequestRetryPolicy.java:261)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0.call()
local variable: com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0#2
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:62)
local variable: reactor.core.publisher.MonoCallable#16
local variable: java.time.Duration#229
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:55)
local variable: reactor.core.publisher.MonoCallable#16
at com.azure.storage.common.policy.RequestRetryPolicy.attemptSync(RequestRetryPolicy.java:270)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.HttpRequest#4
local variable: reactor.core.publisher.MonoCallable#16
at com.azure.storage.common.policy.RequestRetryPolicy.processSync(RequestRetryPolicy.java:71)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.HttpRequest#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.storage.common.policy.RequestRetryPolicy#6
at com.azure.core.http.policy.RequestIdPolicy.processSync(RequestIdPolicy.java:77)
local variable: com.azure.core.http.policy.RequestIdPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.policy.RequestIdPolicy#6
at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
local variable: com.azure.core.http.policy.UserAgentPolicy$1#8
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
local variable: com.azure.core.http.policy.UserAgentPolicy#8
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.policy.UserAgentPolicy#8
at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
local variable: com.azure.core.http.HttpPipeline#7
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#32
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#32
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
local variable: com.azure.core.util.tracing.Utils$$Lambda+0x000078380135f268#1
at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#32
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: java.lang.Object[]#9605
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
local variable: com.azure.core.http.rest.RestProxy#32
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.reflect.Method#8034
local variable: java.lang.Object[]#9605
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.util.Context$1#1
at jdk.proxy2.$Proxy213.listBlobFlatSegmentSync()
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.String#182829
local variable: java.lang.String#102455
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#39623
local variable: java.lang.String#102453
local variable: java.lang.String#139122
local variable: com.azure.core.util.Context$1#1
at com.azure.storage.blob.implementation.ContainersImpl.listBlobFlatSegmentWithResponse(ContainersImpl.java:5959)
local variable: com.azure.storage.blob.implementation.ContainersImpl#6
local variable: java.lang.String#102455
local variable: java.lang.String#39623
local variable: com.azure.core.util.Context$1#1
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#139122
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$6(BlobContainerClient.java:1046)
local variable: com.azure.storage.blob.BlobContainerClient#6
local variable: com.azure.storage.blob.models.ListBlobsOptions#3
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760.call()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#2
at com.azure.storage.common.implementation.StorageImplUtils.sendRequest(StorageImplUtils.java:494)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#2
local variable: class com.azure.storage.blob.models.BlobStorageException
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$7(BlobContainerClient.java:1050)
local variable: com.azure.storage.blob.BlobContainerClient#6
local variable: com.azure.storage.blob.models.ListBlobsOptions#4
local variable: com.azure.storage.blob.models.ListBlobsOptions#3
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#2
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#2
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$8(BlobContainerClient.java:1065)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#2
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#2
at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#2
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#2
at com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40.getPage()
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#2
at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#2
local variable: java.util.concurrent.atomic.AtomicBoolean#271
at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.(ContinuablePagedByItemIterable.java:83)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#2
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#2
local variable: com.azure.core.http.rest.PagedIterableBase$$Lambda+0x00007838015367b0#1
at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:58)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable#2
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#2
at java.lang.Iterable.spliterator()
at com.azure.core.util.paging.ContinuablePagedIterable.stream(ContinuablePagedIterable.java:85)
local variable: com.azure.core.http.rest.PagedIterable#2
The heap dump shows multiple virtual threads. `azure-sdk-global-thread-5` and `azure-sdk-global-thread-4` are started but don't have a carrier thread yet. Thread `#415` and `#416` are the threads from above that are pinned to their carrier threads.
**To Reproduce**
Steps to reproduce the behavior:
- Have virtual threads available
- Have few CPUs available, leading to a small number of carrier threads
- Have multiple `DefaultAzureCredential` instances
- Make multiple concurrent request that need to retrieve a new token from the credential chain
This is the scenario that leads to fairly reliable deadlocks. I'm not actually sure if multiple `DefaultAzureCredential` instances are required or not, or even if the issue would also occur when using one of the authentication methods explicitly.
**Expected behavior**
No deadlocks. The Azure SDK should not use virtual threads if it pins them by using `synchronized` or calling into native code.
**Setup (please complete the following information):**
- OS: Linux
- Library/Libraries: Azure SDK 1.2.36
- Java version: 21
**Information Checklist**
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report
- [x] Bug Description Added
- [x] Repro Steps Added
- [x] Setup information Added
Contributor guide
Assessment
This issue has not been assessed yet.