Azure / Azure/azure-sdk-for-python

Cosmos: Per-Partition Circuit Breaker should only be enabled when multiple regions are available

Open
#46,327 0 comments 0 reactions 0 assignees View on GitHub
Client Cosmos
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

## Description

The per-partition circuit breaker (PPCB) should only be enabled when there are multiple regions a request can be routed to. If there is only a single region available, circuit breaking provides no benefit since there is no fallback region to route to.

## Current Behavior

In [`_global_partition_endpoint_manager_circuit_breaker_core.py` (L60-L85)](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cosmos/azure-cosmos/azure/cosmos/_global_partition_endpoint_manager_circuit_breaker_core.py#L60-L85), `is_circuit_breaker_applicable` checks whether multi-write locations can be used for write requests, but does not verify that there are actually multiple regions available for the request type (read or write).

For example, if a client has only 1 preferred region or excludes all but 1 region, PPCB should not kick in since there is nowhere to failover to.

## Expected Behavior

Add a check similar to the Java SDK implementation in [`GlobalPartitionEndpointManagerForPerPartitionCircuitBreaker.java` (L429-L442)](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/GlobalPartitionEndpointManagerForPerPartitionCircuitBreaker.java#L429-L442):

- If multi-write locations **cannot** be used:
- For **write** requests: return `False` (already handled)
- For **read** requests: only enable PPCB if there are **more than 1** applicable read endpoints
- If multi-write locations **can** be used:
- Only enable PPCB if there are **more than 1** applicable write endpoints

### Java SDK Reference

```java
if (!globalEndpointManager.canUseMultipleWriteLocations(request)) {
if (!request.isReadOnlyRequest()) {
return false;
}
UnmodifiableList applicableReadEndpoints =
globalEndpointManager.getApplicableReadRegionalRoutingContexts(Collections.emptyList());
return applicableReadEndpoints != null && applicableReadEndpoints.size() > 1;
}

UnmodifiableList applicableWriteEndpoints =
globalEndpointManager.getApplicableWriteRegionalRoutingContexts(Collections.emptyList());
return applicableWriteEndpoints != null && applicableWriteEndpoints.size() > 1;
```

## Suggested Change

In `is_circuit_breaker_applicable`, after the existing write-location check, add logic to verify the number of applicable endpoints:

1. When `can_use_multiple_write_locations` is `False` and the request is a read: check that the number of applicable read regions > 1
2. When `can_use_multiple_write_locations` is `True`: check that the number of applicable write regions > 1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.