Azure / Azure/azure-sdk-for-python
[Cosmos] Metadata retry: cross-region failover preempted when caller cancels during control-plane timeout escalation
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
## Summary
Same class of defect as Azure/azure-cosmos-dotnet-v3#5805 (fix in Azure/azure-cosmos-dotnet-v3#5806).
When a control-plane metadata read (e.g., container cache warm-up via `ReadContainer`) hits an unhealthy preferred region, the SDK escalates through internal HTTP timeouts and ultimately receives a 503 (or times out). The client retry policy would normally route the next attempt to the next preferred region, but if the caller's request-level timeout / cancellation fires during the escalation, the retry loop exits with `CancelledError` (async) or returns out of the `while True` (sync) **before** `retry_policy.ShouldRetry` is consulted.
Net effect: a customer who configures a modest per-request timeout never gets the cross-region failover that the SDK advertises for cold-cache metadata reads.
## Files / lines
- `sdk/cosmos/azure-cosmos/azure/cosmos/_retry_utility.py` — sync retry loop (~line 187)
- `sdk/cosmos/azure-cosmos/azure/cosmos/aio/_retry_utility_async.py` — async retry loop (~line 187)
Shape of the buggy loop:
```python
while True:
try:
return await operation() # or operation() for sync
except exceptions.CosmosHttpResponseError as e:
retry_result = retry_policy.ShouldRetry(e)
if not retry_result:
raise
...
```
`asyncio.CancelledError` / `BaseException` subclasses raised by the caller's cancellation bypass the `except CosmosHttpResponseError` and propagate out before the policy runs.
## Repro (sketch)
1. Fresh `CosmosClient` (cold container cache).
2. Configure preferred regions `[A, B]` where `A` is unreachable (blackhole / 503).
3. Issue `container.query_items(...)` (or `read_item`) with a caller-level timeout of ~36s (roughly the worst-case HTTP escalation window: 0.5s + 5s + 30s).
4. Observe: caller receives `CancelledError` / timeout; SDK makes **no** attempt against region `B`.
With region `A` healthy, or with the cache already warm, the code path is not exercised and the call succeeds — which is why this reproduces only on cold-cache + region-failure.
## Expected behavior
The retry policy is consulted even when the caller's cancellation has fired during the in-flight metadata call; if the policy would have routed the retry to the next region, one bounded cross-region attempt executes (on a cancellation-shielded scope) before the original failure is surfaced to the caller.
## Proposed fix direction
- Catch `BaseException` (sync) or `asyncio.CancelledError` (async) once per loop iteration alongside `CosmosHttpResponseError`.
- Consult `retry_policy.ShouldRetry` on the captured exception.
- If the policy indicates a cross-region retry, run one attempt against a fresh, bounded cancellation scope:
- async: `await asyncio.wait_for(asyncio.shield(op(...)), timeout=grace_seconds)` (e.g., 10s).
- sync: run on a detached thread with a bounded join.
- On success, return; on failure / grace expiry, re-raise the original exception via `raise ... from e` (preserve the original as the user-visible cause).
The .NET fix (#5806) uses this pattern with a 10s default grace window; matching that is a reasonable starting point.
## Cross-references
This defect class was identified in a cross-SDK investigation prompted by the .NET fix. Tracking issues:
- .NET (fix landed): Azure/azure-cosmos-dotnet-v3#5805 — PR Azure/azure-cosmos-dotnet-v3#5806
- Rust: Azure/azure-sdk-for-rust#4253
- Go: Azure/azure-sdk-for-go#26649
- Java: not filed — Reactor's `retryWhen` operator structurally isolates subscription cancellation from the retry decision, so the defect does not reproduce. If a repro surfaces, file a follow-up.
/cc @NaluTripician
Contributor guide
Assessment
This issue has not been assessed yet.