[BUG] Slow VisualStudioCredential performance when accessing Azure App Config in a multi-tenant setup

# Issue Transfer

This issue has been transferred from the Azure SDK for .NET repository, [#41433](https://github.com/Azure/azure-sdk-for-net/issues/41433).

**AppConfig Team:** This appears to be related to the AppConfig extension and not the credential itself, as other uses of the same credential - such as KeyVault access - do not have suffer the same slow performance. If information on the Azure.Identity credentials is needed, please reach out internally for discussion.

### Please be aware that @clbarrett is the author of the original issue and include them for any questions or replies.

## Details

### Library name and version

Microsoft.Azure.AppConfiguration.AspNetCore 7.0.0
Azure.Identity 1.10.4

### Describe the bug

The situation is that I'm setting up a ChainedTokenCredential to use the Visual Studio Credentials followed by Azure CLI Credentials when running locally. Weirdly I'm seeing a connection time between 15-45 seconds when calling builder.Configuration.AddAzureAppConfiguration(...) using the Visual Studio Credential. But the connection is less than a second when using Azure CLI alone. Same behavior persists when I used only Visual Studio Credentials. Neither do I have the same issue connecting to an Azure Key Vault. Do note that this is in a multi-tenant scenario.

### Expected behavior

Expect that the call to builder.Configuration.AddAzureAppConfiguration(...) completes in less than a second, just like when adding a Key Vault or using the Azure CLI Credentials.

### Actual behavior

The builder.Configuration.AddAzureAppConfiguration(...) call takes anywhere from 15 to 45 seconds to complete.

### Reproduction Steps

My user is on Tenant A while my App Config is on Tenant B. The Tenant ID in my code is the Tenant B TenantID.

```csharp
List credentials = new List();
VisualStudioCredentialOptions vscOptions = new VisualStudioCredentialOptions()
{
AdditionallyAllowedTenants = { "*" },
//Not a real Tenant ID
TenantId = "88909005-48a7-48b4-b99b-73b1295c6424",
};
credentials.Add(new VisualStudioCredential(vscOptions));
var AzureIdentity = new ChainedTokenCredential(credentials.ToArray());

config.AddAzureAppConfiguration(options =>
{
options.Connect("https://insertappconfignamehere.azconfig.io/", AzureIdentity)
.Select(KeyFilter.Any, LabelFilter.Null);
});
```

For comparison swap VisualStudioCredential & Options with AzureCliCredential & Options.

### Environment

Windows 11
Visual Studio 2022 64 bit - Version 17.8.3
C#
Asp.Net 6.0