[BUG] Excessive Azure Key Vault API Requests After az-appconfig-k8s-provider Pod Restart (v2.2.4)

Environment:
Azure App Configuration Kubernetes Provider Version: 2.2.4
AKS Version: 1.31.x (tested on 1.31.2–1.31.8)
Authentication Method: Workload Identity
Secret Refresh Interval: 1h
Observed Behavior After Pod Restart: Surge in Azure Key Vault API requests
Configuration Snippet:
```
...
spec:
auth:
workloadIdentity:
serviceAccountName: managedId
configuration:
refresh:
enabled: true
...
secret:
auth:
workloadIdentity:
serviceAccountName: managedId
refresh:
enabled: true
interval: 1h

```
Steps to Reproduce:
1 Remove all existing AzureAppConfigurationProvider custom resources.
2 Deploy AzureAppConfigurationProvider for app1.
3 Deploy AzureAppConfigurationProvider for app2.
4 Deploy AzureAppConfigurationProvider for app3.
5 Restart the az-appconfig-k8s-provider pod manually.


AzureDiagnostics
```
clientInfo_s httpStatusCode_d count_
azsdk-go-azsecrets/v1.3.1 (go1.23.9; linux) 200 144064
azsdk-go-azsecrets/v1.3.1 (go1.23.9; linux) 401 142775
```

Expected Behavior:
After pod restart (during AKS upgrades), secret refresh should continue on the configured 1-hour interval without a spike in Azure Key Vault API requests.