OpenID callback flow hits unauthorized

**Describe the bug**

We are using AAD as our "main" source of auth - however due to MFA restrictions on users in the AAD, we have an OpenID Connect server (IdentityServer) for the purpose of test users.
Lately we have observed that the last step of the OAuth flow, the callback, will redirect to initiate the AAD login flow instead (as if the request is unauthorized).
The weird part is that this is sporadic so sometimes it is successful and the user is authorized but many times it is not.

**To Reproduce**
The SWA is not public but I'll try to explain the flow here (can provide a HAR dump if necessary).
1. GET https://REDACTED.azurestaticapps.net/.auth/login/test
2. GET https://REDACTED.azurestaticapps.net/.auth/login/test?post_login_redirect_uri=/.auth/complete&staticWebAppsAuthNonce=ItcUM53Zb721QqABRHVqCYcr7pKr64UIrMebXlxHP8I1OsXEaiHsiethFd5VjmupAO3P2laEg88N7BZ3%2b1kObJeYqiFno%2b7K0GYa7%2bdBW63Fk%2bX%2bAONg3Lpmo7H1nweD
3. FORWARDS to OpenID Server: https://app-REDACTED-us-e-1.azurewebsites.net/connect/authorize?response_type=code&client_id=e2etestclientid&redirect_uri=https%3A%2F%2FREDACTED.azurestaticapps.net%2F.auth%2Flogin%2Ftest%2Fcallback&nonce=21b3726e89de49ceb7c269224910214e_20231019073216&state=redir%3D%252F.auth%252Fcomplete&scope=openid+profile+roles+email&post_login_redirect_uri=%2F.auth%2Fcomplete&staticWebAppsAuthNonce=ItcUM53Zb721QqABRHVqCYcr7pKr64UIrMebXlxHP8I1OsXEaiHsiethFd5VjmupAO3P2laEg88N7BZ3%2B1kObJeYqiFno%2B7K0GYa7%2BdBW63Fk%2BX%2BAONg3Lpmo7H1nweD
4. Enter login info and submit
5. POST login info to https://app-REDACTED-us-e-1.azurewebsites.net/Account/Login?ReturnUrl=%2Fconnect%2Fauthorize%2Fcallback%3Fresponse_type%3Dcode%26client_id%3De2etestclientid%26redirect_uri%3Dhttps%253A%252F%252FREDACTED.azurestaticapps.net%252F.auth%252Flogin%252Ftest%252Fcallback%26nonce%3D21b3726e89de49ceb7c269224910214e_20231019073216%26state%3Dredir%253D%25252F.auth%25252Fcomplete%26scope%3Dopenid%2520profile%2520roles%2520email%26post_login_redirect_uri%3D%252F.auth%252Fcomplete%26staticWebAppsAuthNonce%3DItcUM53Zb721QqABRHVqCYcr7pKr64UIrMebXlxHP8I1OsXEaiHsiethFd5VjmupAO3P2laEg88N7BZ3%252B1kObJeYqiFno%252B7K0GYa7%252BdBW63Fk%252BX%252BAONg3Lpmo7H1nweD
6. GET https://app-REDACTED-us-e-1.azurewebsites.net/connect/authorize/callback?response_type=code&client_id=e2etestclientid&redirect_uri=https%3A%2F%2FREDACTED.azurestaticapps.net%2F.auth%2Flogin%2Ftest%2Fcallback&nonce=21b3726e89de49ceb7c269224910214e_20231019073216&state=redir%3D%252F.auth%252Fcomplete&scope=openid%20profile%20roles%20email&post_login_redirect_uri=%2F.auth%2Fcomplete&staticWebAppsAuthNonce=ItcUM53Zb721QqABRHVqCYcr7pKr64UIrMebXlxHP8I1OsXEaiHsiethFd5VjmupAO3P2laEg88N7BZ3%2B1kObJeYqiFno%2B7K0GYa7%2BdBW63Fk%2BX%2BAONg3Lpmo7H1nweD
7. GET https://REDACTED.azurestaticapps.net/.auth/login/test/callback?code=956C354430285E7D5E58899B379ED8EADDBCAEE9014E38DA7A27A2936B2C51F3&scope=openid%20profile%20roles%20email&state=redir%3D%252F.auth%252Fcomplete&session_state=DcTlyhBYxdrlzzjOUvm-bUwZUUNKhsZK4aQDxNKUWro.E54146CE19A619491F62BA3FD00A2A3F

But step # 7 sets a Location header to
/.auth/login/aad?post_login_redirect_uri=/.auth/login/test/callback?code%3d956C354430285E7D5E58899B379ED8EADDBCAEE9014E38DA7A27A2936B2C51F3%26scope%3dopenid%2520profile%2520roles%2520email%26state%3dredir%253D%25252F.auth%25252Fcomplete%26session_state%3dDcTlyhBYxdrlzzjOUvm-bUwZUUNKhsZK4aQDxNKUWro.E54146CE19A619491F62BA3FD00A2A3F

```json
{
"routes": [
{
"route": "/api/health",
"allowedRoles": ["anonymous"]
},
{
"route": "/logout",
"redirect": "/.auth/logout"
},
{
"route": "/*",
"allowedRoles": ["access"],
"headers": {
"Cache-Control": "no-store"
}
}
],
"navigationFallback": {
"rewrite": "index.html",
"exclude": ["*.{css,scss,js,png,gif,ico,jpg,svg}"]
},
"responseOverrides": {
"401": {
"statusCode": 302,
"redirect": "/.auth/login/aad?post_login_redirect_uri=.referrer"
}
},
"auth": {
"rolesSource": "/api/GetRoles",
"identityProviders": {
"azureActiveDirectory": {
"registration": {
"openIdIssuer": "https://login.microsoftonline.com/REDACTED/v2.0",
"clientIdSettingName": "AZURE_CLIENT_ID",
"clientSecretSettingName": "AZURE_CLIENT_SECRET"
},
"login": {
"loginParameters": []
}
},
"customOpenIdConnectProviders": {
"test": {
"registration": {
"clientIdSettingName": "OPENID_CLIENT_ID",
"clientCredential": {
"clientSecretSettingName": "OPENID_CLIENT_SECRET"
},
"openIdConnectConfiguration": {
"wellKnownOpenIdConfiguration": "https://app-REDACTED-dev-us-e-1.azurewebsites.net/.well-known/openid-configuration"
}
},
"login": {
"nameClaimType": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name",
"scopes": [
"openid",
"profile",
"roles",
"email"
],
"loginParameterNames": []
}
}
}
}
}
}
```

**Expected behavior**
The user to be authorized and taken to the application