OpenID Metadata not getting cached

After upgrading my .net 8 Project to .net 9 or .net 10 I have issues with OpenID Metadata not getting cached when validating an Entra Token.
Each request the OpenID data get fetched again.

Performance **before uprading**:

**After upgrading** to newer .net including new Authentication.JWTBearer Package

**Before Upgrade:**

Request to /usageTypes took 139ms.
Breakdown: PreAuth: 0,05ms,
Validation: 0,16ms,
Endpoint: 139,38ms.

**After Upgrade**
Request to /usageTypes took 456ms.
Breakdown: PreAuth: 30,96ms,
Validation: 248,30ms,
Endpoint: 176,93ms

My program.cs configuration is very basic. Nothing special:

I can fix it by either downgrading to .net 8 or precaching the configManager:

The endpoint is also very simple

### To Reproduce
In my case, just use any entra configuration with

`builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAd"));`

I tested it on different devices (same issue).

` "AzureAd": {
"Instance": "https://login.microsoftonline.com/",
"TenantId": "xxxxxx",
"ClientId": "yyyyy",
"CallbackPath": "/signin-oidc"
},`

### Further technical details
details of dotnet --info

.NET SDKs installed:
9.0.306 [C:\Program Files\dotnet\sdk]
10.0.102 [C:\Program Files\dotnet\sdk]
10.0.103 [C:\Program Files\dotnet\sdk]

- Rider, Windows 11.
My csproj: