dotnet / dotnet/arcade

Handle HTTPClient Timeout (TaskCancelledException) in Arcade Helix SDKs

Open
#2,826 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
729
Forks
397
Avg merge
3d 15m
Merged PRs (30d)
149

Description

Please resolve as dupe if this already exists. The problem may actually be in Microsoft.Rest.RetryDelegatingHandler (it's pretty common to catch all HttpRequestExceptions and ignore the TaskCanceleled ones)

Seen in the logs of this run:
https://dev.azure.com/dnceng/public/_build/results?buildId=192431&view=logs

```
2019-05-17T20:30:56.7945468Z Stopping Azure Pipelines Test Run Linux arm64 Checked @ (Alpine.38.Arm64.Open)Ubuntu.1604.Arm64.Docker.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.8-helix-arm64v8-46e69dd-20190313223330
2019-05-17T20:33:39.2480228Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : TaskCanceledException: The operation was canceled. [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2484166Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2487095Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2489735Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2492262Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2494814Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.Rest.RetryAfterDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2497670Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.Rest.RetryDelegatingHandler.<>c__DisplayClass11_0.<b__1>d.MoveNext() [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2500622Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : --- End of stack trace from previous location where exception was thrown --- [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2503106Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2505546Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2508106Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Client.Aggregate.JobSummaryInternalAsync(IImmutableList`1 groupBy, Int32 maxResultSets, String filterBuild, String filterCreator, String filterName, String filterSource, String filterType, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Aggregate.cs:line 581 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2511225Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Client.Aggregate.JobSummaryAsync(IImmutableList`1 groupBy, Int32 maxResultSets, String filterBuild, String filterCreator, String filterName, String filterSource, String filterType, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Aggregate.cs:line 493 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2514621Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.MissionControlTestProcessingDoneAsync(String jobName) in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 68 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2516412Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.CheckHelixJobAsync(String jobName) in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 56 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2517757Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.ExecuteCore() in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 28 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2519034Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 43 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2520604Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : IOException: Unable to read data from the transport connection: Operation canceled. [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2521656Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2545316Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2546747Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Security.SslStreamInternal.g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2548324Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2550044Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2551213Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : SocketException: Operation canceled [/__w/1/s/tests/helixpublishwitharcade.proj]

```

Contributor guide

Open the contributing guide

Research direction

Start with Microsoft.Rest.RetryDelegatingHandler and the call path through Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.cs, especially the references at lines 56 and 68. Reproduce or trace the failure from tests/helixpublishwitharcade.proj and inspect the generated Aggregate.cs calls; done means the HTTP timeout or cancellation is handled without the Helix task failing unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.