Azure / Azure/azure-sdk-for-python

[Cosmos] Metadata retry: cross-region failover preempted when caller cancels during control-plane timeout escalation

Abierto
#46,471 0 comentarios 0 reacciones 1 asignado Reclamado por @NaluTripician Ver en GitHub
Lenguaje dominante
Python
Estrellas
5.6k
Forks
3.4k
Merge medio
2 d 2 h
PR fusionados (30 d)
213

Descripción

## Summary

Same class of defect as Azure/azure-cosmos-dotnet-v3#5805 (fix in Azure/azure-cosmos-dotnet-v3#5806).

When a control-plane metadata read (e.g., container cache warm-up via `ReadContainer`) hits an unhealthy preferred region, the SDK escalates through internal HTTP timeouts and ultimately receives a 503 (or times out). The client retry policy would normally route the next attempt to the next preferred region, but if the caller's request-level timeout / cancellation fires during the escalation, the retry loop exits with `CancelledError` (async) or returns out of the `while True` (sync) **before** `retry_policy.ShouldRetry` is consulted.

Net effect: a customer who configures a modest per-request timeout never gets the cross-region failover that the SDK advertises for cold-cache metadata reads.

## Files / lines

- `sdk/cosmos/azure-cosmos/azure/cosmos/_retry_utility.py` — sync retry loop (~line 187)
- `sdk/cosmos/azure-cosmos/azure/cosmos/aio/_retry_utility_async.py` — async retry loop (~line 187)

Shape of the buggy loop:

```python
while True:
try:
return await operation() # or operation() for sync
except exceptions.CosmosHttpResponseError as e:
retry_result = retry_policy.ShouldRetry(e)
if not retry_result:
raise
...
```

`asyncio.CancelledError` / `BaseException` subclasses raised by the caller's cancellation bypass the `except CosmosHttpResponseError` and propagate out before the policy runs.

## Repro (sketch)

1. Fresh `CosmosClient` (cold container cache).
2. Configure preferred regions `[A, B]` where `A` is unreachable (blackhole / 503).
3. Issue `container.query_items(...)` (or `read_item`) with a caller-level timeout of ~36s (roughly the worst-case HTTP escalation window: 0.5s + 5s + 30s).
4. Observe: caller receives `CancelledError` / timeout; SDK makes **no** attempt against region `B`.

With region `A` healthy, or with the cache already warm, the code path is not exercised and the call succeeds — which is why this reproduces only on cold-cache + region-failure.

## Expected behavior

The retry policy is consulted even when the caller's cancellation has fired during the in-flight metadata call; if the policy would have routed the retry to the next region, one bounded cross-region attempt executes (on a cancellation-shielded scope) before the original failure is surfaced to the caller.

## Proposed fix direction

- Catch `BaseException` (sync) or `asyncio.CancelledError` (async) once per loop iteration alongside `CosmosHttpResponseError`.
- Consult `retry_policy.ShouldRetry` on the captured exception.
- If the policy indicates a cross-region retry, run one attempt against a fresh, bounded cancellation scope:
- async: `await asyncio.wait_for(asyncio.shield(op(...)), timeout=grace_seconds)` (e.g., 10s).
- sync: run on a detached thread with a bounded join.
- On success, return; on failure / grace expiry, re-raise the original exception via `raise ... from e` (preserve the original as the user-visible cause).

The .NET fix (#5806) uses this pattern with a 10s default grace window; matching that is a reasonable starting point.

## Cross-references

This defect class was identified in a cross-SDK investigation prompted by the .NET fix. Tracking issues:

- .NET (fix landed): Azure/azure-cosmos-dotnet-v3#5805 — PR Azure/azure-cosmos-dotnet-v3#5806
- Rust: Azure/azure-sdk-for-rust#4253
- Go: Azure/azure-sdk-for-go#26649
- Java: not filed — Reactor's `retryWhen` operator structurally isolates subscription cancellation from the retry decision, so the defect does not reproduce. If a repro surfaces, file a follow-up.

/cc @NaluTripician

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.