Azure / Azure/azure-sdk-for-python
Cosmos: Per-Partition Circuit Breaker should only be enabled when multiple regions are available
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
## Description
The per-partition circuit breaker (PPCB) should only be enabled when there are multiple regions a request can be routed to. If there is only a single region available, circuit breaking provides no benefit since there is no fallback region to route to.
## Current Behavior
In [`_global_partition_endpoint_manager_circuit_breaker_core.py` (L60-L85)](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cosmos/azure-cosmos/azure/cosmos/_global_partition_endpoint_manager_circuit_breaker_core.py#L60-L85), `is_circuit_breaker_applicable` checks whether multi-write locations can be used for write requests, but does not verify that there are actually multiple regions available for the request type (read or write).
For example, if a client has only 1 preferred region or excludes all but 1 region, PPCB should not kick in since there is nowhere to failover to.
## Expected Behavior
Add a check similar to the Java SDK implementation in [`GlobalPartitionEndpointManagerForPerPartitionCircuitBreaker.java` (L429-L442)](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/perPartitionCircuitBreaker/GlobalPartitionEndpointManagerForPerPartitionCircuitBreaker.java#L429-L442):
- If multi-write locations **cannot** be used:
- For **write** requests: return `False` (already handled)
- For **read** requests: only enable PPCB if there are **more than 1** applicable read endpoints
- If multi-write locations **can** be used:
- Only enable PPCB if there are **more than 1** applicable write endpoints
### Java SDK Reference
```java
if (!globalEndpointManager.canUseMultipleWriteLocations(request)) {
if (!request.isReadOnlyRequest()) {
return false;
}
UnmodifiableList applicableReadEndpoints =
globalEndpointManager.getApplicableReadRegionalRoutingContexts(Collections.emptyList());
return applicableReadEndpoints != null && applicableReadEndpoints.size() > 1;
}
UnmodifiableList applicableWriteEndpoints =
globalEndpointManager.getApplicableWriteRegionalRoutingContexts(Collections.emptyList());
return applicableWriteEndpoints != null && applicableWriteEndpoints.size() > 1;
```
## Suggested Change
In `is_circuit_breaker_applicable`, after the existing write-location check, add logic to verify the number of applicable endpoints:
1. When `can_use_multiple_write_locations` is `False` and the request is a read: check that the number of applicable read regions > 1
2. When `can_use_multiple_write_locations` is `True`: check that the number of applicable write regions > 1
Contributor guide
Assessment
This issue has not been assessed yet.