Azure / Azure/azure-sdk-for-rust
[Cosmos] No concurrency budget bounds hedge request amplification
- Dominant language
- Rust
- Stars
- 884
- Forks
- 365
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 112
Description
## Context
Measured while building the fault-injection stress harness for cross-region metadata hedging (#4896). Full numbers are in [this PR comment](https://github.com/Azure/azure-sdk-for-rust/pull/4896#issuecomment-5122932465).
## Problem
When the primary region's metadata path browns out, every eligible metadata read that crosses the 1.5 s threshold dispatches a second request to the alternate region. Measured amplification under sustained pressure:
| Scenario | metadata requests OFF → ON | amplification |
|---|---|---|
| Refresh, low contention (15 sequential) | 15 → 30 | 2.00× |
| Saturating storm (32 concurrent) | 32 → 64 | 2.00× |
| Cold start (8 iterations) | 24 → 40 | 1.67× |
| Mixed 70/30 (30 ops) | 12 → 23 | 1.92× |
There is no per-client hedge concurrency budget today, so nothing caps how many alternate-region requests can be in flight at once. A wide enough metadata fan-out during a regional brownout sends the full doubled volume at the healthy region — precisely when that region is also absorbing failover traffic.
In practice this is bounded by how many *distinct* metadata reads can be outstanding (the container and PK-range caches coalesce the rest), which is why it did not show up as a problem in the harness. But the bound is incidental rather than designed.
## Prior art
.NET raised the same tradeoff in [Azure/azure-cosmos-dotnet-v3#5999](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5999), where removing the per-client concurrency budget was called out explicitly as the cost of the latency win.
## Suggested direction
Some form of admission control on concurrent hedge dispatches — a semaphore-style budget, or a cheaper adaptive signal (e.g. stop hedging once the alternate region's in-flight count crosses a threshold, since a hedge that queues behind other hedges has already lost its latency argument).
Worth deciding whether metadata hedging shares a budget with data-plane hedging or gets its own; they have very different fan-out characteristics.
## Repro
```text
RUN_HEDGE_STRESS=1 cargo test -p azure_data_cosmos_driver \
--features fault_injection,__internal_in_memory_emulator \
--test in_memory_emulator metadata_hedging_stress -- --ignored --nocapture
```
The harness reports amplification per scenario; `HEDGE_STRESS_STORM_CONCURRENCY` controls the storm fan-out.
Contributor guide
Research direction
Start with the in-memory emulator test entry point `metadata_hedging_stress` and run the provided `RUN_HEDGE_STRESS=1 cargo test` command with the listed features. Review the harness results and the existing metadata and data-plane hedging paths to decide the concurrency-budget behavior. Done means an agreed admission-control design is implemented and the stress scenarios verify bounded amplification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100