Pits of failure using `WaitForResourceHealthyAsync` in tests.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Is your feature request related to a problem? Please describe the problem.
In tests, it is often useful to wait for a resource to be healthy before progressing with your test. However if the health checks fail during tests / CI, it can be a pain to troubleshoot them. For local debugging #13550 will help, but that won't help in CI, or if running tests without a debugger.
Take the following example - this starts a simple nginx container, with two health checks that will never go healthy. In it's current form, the test will hang forever, with little diagnostic information to know why it's hanging.
```cs
[Fact]
public async Task ResourceDoesntGoHealthy()
{
var cts = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken);
await using var builder = DistributedApplicationTestingBuilder.Create();
//cts.CancelAfter(10_000);
//builder.Services.AddLogging(x => x.AddFilter("Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService", LogLevel.Information));
var nginx = builder.AddContainer("nginx", "nginx")
.WithHttpEndpoint(targetPort: 80)
.WithHttpHealthCheck("/does-not-exist");
await using var app = builder.Build();
await app.StartAsync(cts.Token);
await app.ResourceNotifications.WaitForResourceHealthyAsync(nginx.Resource.Name, cts.Token);
}
```
### Describe the solution you'd like
The first two pits of failure you encounter are:
1. When resource fails to go healthy, it is very easy to write your test in such a way that it'll deadlock. (i.e. forgetting to flow cancellation tokens appropriately)
2. If you do remember/know to add an explicit timeout, then you'll get back no logs about why the health check failed, and the exception will jut be an unhelpful "The operation was cancelled."
To help this, I see a few things being necessary:
### 1. Default Timeout when waiting for healthy:
We should support a default timeout on waiting for healthy operations. If a resource fails to go healthy within this time, the resource should be considered `FailedToStart`. When the dashboard is in use, it would default to infinite. But when used without the dashboard, or in a testing scenario, it should have some sanity default (say 5/10 minutes), to cause the tests to fail fast if something doesn't happen in time. This value should be user configurable, with that value superseding the defaults.
(Related to #5633)
### 2. Don't supress `DefaultHealthCheckService` logs when Dashboard not present
`DistributedApplicationBuilder` currently forces the `Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService` log level to `None`. Whilst this is reasonable to do when the dashboard is available, doing this in tests means you loose all ability to see why health checks are failing. I would change this to only disabling these logs when the dashboard is used.
https://github.com/dotnet/aspire/blob/28f530255626ca260d98a226f4626a9494e2913a/src/Aspire.Hosting/DistributedApplicationBuilder.cs#L197-L199
Changing this will cause the following errors to show up in the logs, giving you a chance to see why health checks are failing. (You can simulate this in the above test by uncommenting the `AddLogging()` call.
```
fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
Health check nginx_http_/does-not-exist_200_check with status Unhealthy completed after 38.8117ms with message 'Discover endpoint #0 is not responding with code in 200...200 range, the current status is NotFound.'
fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
Health check nginx_fake_/_200_check with status Unhealthy completed after 7880.4196ms with message '(null)'
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<b__3_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.ThrowIfException()
at Microsoft.Extensions.Http.Resilience.ResilienceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at HealthChecks.Uris.UriHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /home/runner/work/AspNetCore.Diagnostics.HealthChecks/AspNetCore.Diagnostics.HealthChecks/src/HealthChecks.Uris/UriHealthCheck.cs:line 54
```
### 3. Provide more details when `ResourceNotificationService.WaitForXYZ` are cancelled
Whilst the above change does let you see health checks are failing, you do have to trawl through logs to find them. It would be more useful if the error could explicitly highlight the current state of the resource to let you better see why the wait failed.
To aid with this, I've written a wrapper method around `WaitForResourceHealthyAsync` which wraps any thrown `OperationCanceledException` exceptions to add more details to the message about the current sate of the resource. It would be helpful if all of the `ResourceNotificationService.WaitForXYZ` methods did something similar natively.
```cs
async Task WaitForResourceHealthyAsyncBetter(IResourceBuilder builder)
{
try
{
await app.ResourceNotifications.WaitForResourceHealthyAsync(nginx.Resource.Name, cts.Token);
}
catch (OperationCanceledException ex)
{
var resource = builder.Resource;
if (app.ResourceNotifications.TryGetCurrentState(resource.Name, out var evt) && evt.Snapshot != null)
{
var state = evt.Snapshot;
var error = new StringBuilder()
.AppendLine($"Resource {resource.Name} failed to become healthy before WaitForResourceHealthyAsync was cancelled")
.AppendLine($"Current State: {state.State?.Text}")
.AppendLine($"Current Health: {state.HealthStatus}");
foreach(var report in evt.Snapshot.HealthReports)
{
error.AppendLine($"- {report.Name}: {report.Status} @ {report.LastRunAt} {report.ExceptionText}");
}
throw new OperationCanceledException(error.ToString(), ex, ex.CancellationToken);
}
throw new OperationCanceledException($"WaitForResourceHealthyAsync cancelled before resource {nginx.Resource.Name} started", ex, ex.CancellationToken);
}
}
```
With this,
```
System.OperationCanceledException : The operation was canceled.
```
changes to the more useful:
```
System.OperationCanceledException : Resource nginx failed to become healthy before WaitForResourceHealthyAsync was cancelled
Current State: Running
Current Health: Unhealthy
- nginx_http_/does-not-exist_200_check: Unhealthy @ 12/31/2025 10:49:37 AM
- nginx_fake_/_200_check: Unhealthy @ 12/31/2025 10:49:37 AM System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<b__3_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.ThrowIfException()
at Microsoft.Extensions.Http.Resilience.ResilienceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at HealthChecks.Uris.UriHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /home/runner/work/AspNetCore.Diagnostics.HealthChecks/AspNetCore.Diagnostics.HealthChecks/src/HealthChecks.Uris/UriHealthCheck.cs:line 54
---- System.OperationCanceledException : The operation was canceled.
```
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.