TokenEndpoint empty when referencing Microsoft.IdentityModel.Tokens.Saml nuget package

I have some strange behavior when using OpenIdConnect authentication. Everything works fine, but as soon as I reference the Microsoft.IdentityModel.Tokens.Saml 8.0.2 (i also tried older versions) I got the following exception. This exceptions happens on the "way back" from de identity provider (POST on signin-oidc).

```
InvalidOperationException: An invalid request URI was provided. Either the request URI must be an absolute URI or BaseAddress must be set.
System.Net.Http.HttpClient.PrepareRequestMessage(HttpRequestMessage request)

AuthenticationFailureException: An error was encountered while handling the remote login.
Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler.HandleRequestAsync()
```

And after investigating I saw that _configuration?.TokenEndpoint is null, at the below line of code, as soon as I run with a reference to Microsoft.IdentityModel.Tokens.Saml nuget package.

https://github.com/dotnet/aspnetcore/blob/258f8baa683db26ea46a00618a8b9b4d8db764b4/src/Security/Authentication/OpenIdConnect/src/OpenIdConnectHandler.cs#L991

The .csproj with packagerefences looks like this:

```

net8.0
enable
enable
7f837761-f406-4779-ac6c-3f1cbd16fb67

all
runtime; build; native; contentfiles; analyzers; buildtransitive

```

And I found that I need to reference Microsoft.IdentityModel.Tokens.Saml in order for WsFederation to work. So that's the reason that I have reference that package.

Is there something I should configure/override to make this work, or is this a bug?