Azure / Azure/azure-sdk-for-python

[Cosmos] Metadata retry: cross-region failover preempted when caller cancels during control-plane timeout escalation

Open
#46,471 0 comments 0 reactions 1 assignee Claimed by @NaluTripician View on GitHub
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

## Summary

Same class of defect as Azure/azure-cosmos-dotnet-v3#5805 (fix in Azure/azure-cosmos-dotnet-v3#5806).

When a control-plane metadata read (e.g., container cache warm-up via `ReadContainer`) hits an unhealthy preferred region, the SDK escalates through internal HTTP timeouts and ultimately receives a 503 (or times out). The client retry policy would normally route the next attempt to the next preferred region, but if the caller's request-level timeout / cancellation fires during the escalation, the retry loop exits with `CancelledError` (async) or returns out of the `while True` (sync) **before** `retry_policy.ShouldRetry` is consulted.

Net effect: a customer who configures a modest per-request timeout never gets the cross-region failover that the SDK advertises for cold-cache metadata reads.

## Files / lines

- `sdk/cosmos/azure-cosmos/azure/cosmos/_retry_utility.py` — sync retry loop (~line 187)
- `sdk/cosmos/azure-cosmos/azure/cosmos/aio/_retry_utility_async.py` — async retry loop (~line 187)

Shape of the buggy loop:

```python
while True:
try:
return await operation() # or operation() for sync
except exceptions.CosmosHttpResponseError as e:
retry_result = retry_policy.ShouldRetry(e)
if not retry_result:
raise
...
```

`asyncio.CancelledError` / `BaseException` subclasses raised by the caller's cancellation bypass the `except CosmosHttpResponseError` and propagate out before the policy runs.

## Repro (sketch)

1. Fresh `CosmosClient` (cold container cache).
2. Configure preferred regions `[A, B]` where `A` is unreachable (blackhole / 503).
3. Issue `container.query_items(...)` (or `read_item`) with a caller-level timeout of ~36s (roughly the worst-case HTTP escalation window: 0.5s + 5s + 30s).
4. Observe: caller receives `CancelledError` / timeout; SDK makes **no** attempt against region `B`.

With region `A` healthy, or with the cache already warm, the code path is not exercised and the call succeeds — which is why this reproduces only on cold-cache + region-failure.

## Expected behavior

The retry policy is consulted even when the caller's cancellation has fired during the in-flight metadata call; if the policy would have routed the retry to the next region, one bounded cross-region attempt executes (on a cancellation-shielded scope) before the original failure is surfaced to the caller.

## Proposed fix direction

- Catch `BaseException` (sync) or `asyncio.CancelledError` (async) once per loop iteration alongside `CosmosHttpResponseError`.
- Consult `retry_policy.ShouldRetry` on the captured exception.
- If the policy indicates a cross-region retry, run one attempt against a fresh, bounded cancellation scope:
- async: `await asyncio.wait_for(asyncio.shield(op(...)), timeout=grace_seconds)` (e.g., 10s).
- sync: run on a detached thread with a bounded join.
- On success, return; on failure / grace expiry, re-raise the original exception via `raise ... from e` (preserve the original as the user-visible cause).

The .NET fix (#5806) uses this pattern with a 10s default grace window; matching that is a reasonable starting point.

## Cross-references

This defect class was identified in a cross-SDK investigation prompted by the .NET fix. Tracking issues:

- .NET (fix landed): Azure/azure-cosmos-dotnet-v3#5805 — PR Azure/azure-cosmos-dotnet-v3#5806
- Rust: Azure/azure-sdk-for-rust#4253
- Go: Azure/azure-sdk-for-go#26649
- Java: not filed — Reactor's `retryWhen` operator structurally isolates subscription cancellation from the retry decision, so the defect does not reproduce. If a repro surfaces, file a follow-up.

/cc @NaluTripician

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.