Azure / Azure/azure-sdk-for-python
azure-cosmos: async client does not recover from 403 on expired AAD token (bearer token policy only handles 401)
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
## Bug Report
### Description
When using `azure.cosmos.aio.CosmosClient` with `azure.identity.aio.DefaultAzureCredential` (managed identity), the client permanently fails with HTTP 403 after the AAD token expires. Only a full process restart recovers it.
### Root Cause Analysis
The `AsyncBearerTokenCredentialPolicy` in `azure-core` only invalidates its cached `_token` on **HTTP 401** responses ([source](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/azure/core/pipeline/policies/_authentication_async.py#L114)):
```python
if response.http_response.status_code == 401:
self._token = None # any cached token is invalid
```
However, **Cosmos DB returns HTTP 403** (not 401) for expired AAD tokens (sub-status 5300 — `AAD_REQUEST_NOT_AUTHORIZED`).
The `azure-cosmos` retry layer in `_retry_utility_async.py` handles 403 only for region-failover sub-statuses:
- `SubStatusCodes.DATABASE_ACCOUNT_NOT_FOUND` (1008)
- `SubStatusCodes.WRITE_FORBIDDEN` (3)
It does **not** handle 403 with sub-status 5300 (expired/invalid AAD token).
The `AsyncCosmosBearerTokenCredentialPolicy` (in `azure.cosmos.aio._auth_policy_async`) inherits from `AsyncBearerTokenCredentialPolicy` but does not override `send()` to handle 403.
### Result
Once the initial AAD token expires (~1 hour), every subsequent Cosmos request fails with 403. The proactive token refresh (`_should_refresh_token`, 5 minutes before expiry) normally prevents this, but when it fails for any reason (e.g., transient managed identity endpoint unavailability, MSAL cache issues), the SDK has **no recovery path**:
1. The cached `_token` in the policy is expired, but `_should_refresh_token()` calls `credential.get_token()` which may return a cached token from MSAL
2. Cosmos rejects it with 403
3. The policy does not clear `_token` on 403 (only on 401)
4. The cycle repeats indefinitely
### Reproduction
1. Create a long-running application (e.g., FastAPI service on Azure Container Apps)
2. Use `azure.cosmos.aio.CosmosClient` with `azure.identity.aio.DefaultAzureCredential` (managed identity)
3. Follow the recommended singleton pattern (single client instance for the app lifetime)
4. Wait for the AAD token to expire
5. All Cosmos operations fail with 403 and never recover
### Expected Behavior
The `AsyncCosmosBearerTokenCredentialPolicy` should handle HTTP 403 responses with AAD-related sub-statuses by invalidating the cached token and retrying with a fresh one, similar to how the base class handles 401.
### Suggested Fix
In `AsyncCosmosBearerTokenCredentialPolicy`, override `send()` to also clear `self._token` on 403 and retry:
```python
async def send(self, request):
response = await super().send(request)
if response.http_response.status_code == 403:
self._token = None
await self.on_request(request)
response = await self.next.send(request)
return response
```
### Environment
- `azure-cosmos` 4.15.0
- `azure-identity` 1.25.3
- `azure-core` 1.34.0
- Python 3.12
- Runtime: Azure Container Apps with system-assigned managed identity
### Related Issues (other SDKs, same root cause)
- **JavaScript**: https://github.com/Azure/azure-sdk-for-js/issues/22620
- **.NET**: https://github.com/Azure/azure-cosmos-dotnet-v3/issues/3110
This appears to be a cross-SDK design gap — Cosmos DB returns 403 for expired tokens, but all SDK bearer token policies only handle 401.
Contributor guide
Assessment
This issue has not been assessed yet.