OutputCache causes UnobservedTaskException

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

When an exception occurs in a controller method having the OutputCache attribute, an Unobserved Task Exception is thrown when the Garbage Collect is executed.

### Expected Behavior

Expected same behavior on exception as for a method without the OutputCache: no UnobservedTaskException.

### Steps To Reproduce

You can find a small repo to reproduce here: https://github.com/Baars-IT/UnobsTaskExc

- Set a breakpoint in Startup at "e.SetObserved()" to see if an Unobserved Task Exception occurs
- First kick off "/weatherforecast", it will throw an exception, http 500 will be returned.
- Wait till Garbage Collect starts or manually start it via "/g-c"

### Exceptions (if any)

A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The Unobserved Task Exception will occur at Garbage Collection)

InnerException:

"System.ArgumentException: Some exception
at UnobsTaskExc.Controllers.WeatherForecastController.Get() in C:\Data\Temp\UnobservedTaskException\Controllers\WeatherForecastController.cs:line 14
at lambda_method1(Closure, Object, Object[])
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.OutputCaching.OutputCacheMiddleware.<>c__DisplayClass11_0.<g__ExecuteResponseAsync|0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.OutputCaching.WorkDispatcher`2.ScheduleAsync(TKey key, Func`2 valueFactory)
at Microsoft.AspNetCore.OutputCaching.OutputCacheMiddleware.InvokeAwaited(HttpContext httpContext, IReadOnlyList`1 policies)
at Microsoft.WebTools.BrowserLink.Net.BrowserLinkMiddleware.InvokeAsync(HttpContext context)
at Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserRefreshMiddleware.InvokeAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)"

### .NET Version

9.0.203

### Anything else?

Using Visual Studio 17.13.6
ASP.NET Core 9.0.4
Issue occurs for both .NET 8 and .NET 9.
Tried adding custom exception handlers, but issue stays.