AuthenticationMiddleware should prefer the authentication scheme specified in endpoint metadata

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

`AuthenticationMiddleware` should give priority to the authentication scheme defined on the endpoint metadata, rather than always falling back to the default authentication scheme registered at the application level.

This is a very common requirement for applications with multiple authentication schemes: most APIs use the default authentication scheme, while a small subset of endpoints need to use alternative schemes.

It is critical for any custom middleware that runs **between `UseAuthentication()` and `UseAuthorization()`** to get the correct, endpoint‑specific user identity.
With the current implementation, we can observe this problematic flow:

```cs
app.UseAuthentication(); // Sets HttpContext.User = A (incorrect)

// Middlewares running here see User = A

app.UseAuthorization(); // Overwrites HttpContext.User = B (correct)
```
Middleware placed between these two stages will see an incorrect user principal that does not match what the target endpoint actually expects.

I’m aware similar concerns have been raised before. The large number of reports indicates the current API behavior is indeed suboptimal by design. If directly changing the default behavior introduces compatibility risks, an opt‑in AppContext switch could be provided.

Here's a demo for it.
```cs
public class Program
{
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthentication("AuthHandler1")
.AddScheme("AuthHandler1", null)
.AddScheme("AuthHandler2", null);

builder.Services.AddAuthorization();

var app = builder.Build();

app.Use((ctx, next) => { Console.WriteLine("\nBefore UseAuthentication(), User: {0}", ctx.User?.Identity?.Name); return next(); });

app.UseAuthentication();

app.Use((ctx, next) => { Console.WriteLine(" After UseAuthentication(), User: {0}", ctx.User?.Identity?.Name); return next(); });

app.UseAuthorization();

app.Use((ctx, next) => { Console.WriteLine(" After UseAuthorization(), User: {0}", ctx.User?.Identity?.Name); return next(); });

/*
when request "/b", the console outputs:

Before UseAuthentication(), User:
AuthHandler1.HandleAuthenticateAsync...
After UseAuthentication(), User: AuthHandler1 <------------ wrong!
AuthHandler2.HandleAuthenticateAsync...
After UseAuthorization(), User: AuthHandler2
*/

app.Map("/a", (HttpContext ctx) => ctx.User?.Identity?.Name).RequireAuthorization(policy => policy.AddAuthenticationSchemes("AuthHandler1").RequireAuthenticatedUser());
app.Map("/b", (HttpContext ctx) => ctx.User?.Identity?.Name).RequireAuthorization(policy => policy.AddAuthenticationSchemes("AuthHandler2").RequireAuthenticatedUser());
app.Map("/", (HttpContext ctx) => ctx.User?.Identity?.Name);

app.Run();
}

public sealed class AuthHandler(IOptionsMonitor options, ILoggerFactory logger, UrlEncoder encoder)
: AuthenticationHandler(options, logger, encoder)
{
protected override async Task HandleAuthenticateAsync()
{
Console.WriteLine("{0}.HandleAuthenticateAsync...", Scheme.Name);

var principal = new ClaimsPrincipal(new ClaimsIdentity([new Claim(ClaimTypes.Name, Scheme.Name)], Scheme.Name));
return AuthenticateResult.Success(new AuthenticationTicket(principal, Scheme.Name));
}
}
}
```

### Expected Behavior

When a request hits endpoint `/b` which declares `AuthHandler2` as its authentication scheme via authorization policy metadata, `AuthenticationMiddleware` should invoke `AuthHandler2` to authenticate the request during the authentication middleware stage.

### Suggested fix

To address this issue cleanly while maintaining backward compatibility, I propose the following changes:

1. Move Endpoint‑Aware Authentication Logic to AuthenticationMiddleware
Currently, `AuthorizationMiddleware` evaluates the endpoint metadata and invokes `HttpContext.AuthenticateAsync` for the schemes specified in the authorization policy. This logic is misplaced and causes the window between UseAuthentication() and UseAuthorization() to operate on an incorrect HttpContext.User.
I suggest relocating this endpoint‑driven authentication logic entirely into `AuthenticationMiddleware`. This aligns with the Single Responsibility Principle: the authentication middleware should be solely responsible for establishing the user identity based on the target endpoint, while the authorization middleware should focus purely on authorization decisions.

2. Let AuthorizationMiddleware Rely on the Established Result
Once `AuthenticationMiddleware` has correctly set HttpContext.User based on the endpoint’s metadata, `AuthorizationMiddleware` will no longer need to re‑execute authentication. It will simply consume the already‑authenticated principal to evaluate the authorization policies. This ensures that any custom middleware placed between the two stages will see the correct, endpoint‑specific user identity.

3. Introduce Two New Abstraction Interfaces
To make the selection logic extensible and testable, I propose adding the following interfaces:

```csharp
public interface IEffectiveAuthenticationSchemeSelector
{
Task?> SelectEffectiveSchemeAsync(HttpContext context);
}

public interface IEffectiveAuthorizationPolicySelector
{
Task SelectEffectivePolicyAsync(HttpContext context);
}
```
- `IEffectiveAuthenticationSchemeSelector` – Determines which authentication schemes should be challenged/authenticated for the current request, based on the endpoint metadata (or falls back to the default scheme).

- `IEffectiveAuthorizationPolicySelector` – Constructs the effective authorization policy for the current request.

The default implementations will read the authorization policy metadata from the endpoint (just as AuthorizationMiddleware does today), but these abstractions make the behavior customizable and clearly separate the concerns.
```cs
internal sealed class EffectiveAuthorizationPolicySelector
: IEffectiveAuthorizationPolicySelector,
IEffectiveAuthenticationSchemeSelector
{
private readonly IAuthorizationPolicyProvider _policyProvider;
private readonly bool _canCache;
private readonly AuthorizationPolicyCache? _policyCache;

public EffectiveAuthorizationPolicySelector(IAuthorizationPolicyProvider policyProvider, IServiceProvider services)
{
_policyProvider = policyProvider ?? throw new ArgumentNullException(nameof(policyProvider));
if (_policyProvider.AllowsCachingPolicies)
{
ArgumentNullException.ThrowIfNull(services);
_policyCache = services.GetService();
_canCache = _policyCache != null;
}
}

public async Task SelectEffectivePolicyAsync(HttpContext context)
{
// Use the computed policy for this endpoint if we can
AuthorizationPolicy? policy = null;

// Same logic like how AuthorizationMiddleware does today

return policy;
}

public async Task?> SelectEffectiveSchemeAsync(HttpContext context)
{
var policy = await SelectEffectivePolicyAsync(context);
return policy?.AuthenticationSchemes;
}
}
```

4. Preserve Backward Compatibility with an AppContext Switch
Since changing the default behavior could break existing applications that rely on the current flow, I will introduce an opt‑in AppContext switch, e.g.: `Microsoft.AspNetCore.Authentication.UseEndpointAwareAuthentication`.
When the switch is false (default), the middleware pipeline behaves exactly as it does today. When set to true, the new endpoint‑aware authentication flow is enabled, fixing the issue for applications that need it.

I already have a [prototype](https://github.com/heku/aspnetcore/tree/endpoint-aware-authentication) that implements this design. If the team is open to this approach, I will refine the code, add comprehensive tests, and submit a proper pull request. Please let me know your thoughts on this direction so I can decide whether to proceed. I’m happy to adjust the proposal based on your feedback.

Start with the AuthenticationMiddleware and AuthorizationMiddleware entry points, then compare their endpoint-metadata handling with the linked endpoint-aware-authentication prototype. Completion means endpoint metadata selects AuthHandler2 during AuthenticationMiddleware, middleware between the stages sees the endpoint-specific identity, and the current default behavior remains backward compatible with comprehensive tests.

Written by the indexing model from the issue text.