Azure / Azure/azure-sdk-for-rust

[Cosmos] No concurrency budget bounds hedge request amplification

Open
#4,914 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
884
Forks
365
Avg merge
2d 19h
Merged PRs (30d)
112

Description

## Context

Measured while building the fault-injection stress harness for cross-region metadata hedging (#4896). Full numbers are in [this PR comment](https://github.com/Azure/azure-sdk-for-rust/pull/4896#issuecomment-5122932465).

## Problem

When the primary region's metadata path browns out, every eligible metadata read that crosses the 1.5 s threshold dispatches a second request to the alternate region. Measured amplification under sustained pressure:

| Scenario | metadata requests OFF → ON | amplification |
|---|---|---|
| Refresh, low contention (15 sequential) | 15 → 30 | 2.00× |
| Saturating storm (32 concurrent) | 32 → 64 | 2.00× |
| Cold start (8 iterations) | 24 → 40 | 1.67× |
| Mixed 70/30 (30 ops) | 12 → 23 | 1.92× |

There is no per-client hedge concurrency budget today, so nothing caps how many alternate-region requests can be in flight at once. A wide enough metadata fan-out during a regional brownout sends the full doubled volume at the healthy region — precisely when that region is also absorbing failover traffic.

In practice this is bounded by how many *distinct* metadata reads can be outstanding (the container and PK-range caches coalesce the rest), which is why it did not show up as a problem in the harness. But the bound is incidental rather than designed.

## Prior art

.NET raised the same tradeoff in [Azure/azure-cosmos-dotnet-v3#5999](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5999), where removing the per-client concurrency budget was called out explicitly as the cost of the latency win.

## Suggested direction

Some form of admission control on concurrent hedge dispatches — a semaphore-style budget, or a cheaper adaptive signal (e.g. stop hedging once the alternate region's in-flight count crosses a threshold, since a hedge that queues behind other hedges has already lost its latency argument).

Worth deciding whether metadata hedging shares a budget with data-plane hedging or gets its own; they have very different fan-out characteristics.

## Repro

```text
RUN_HEDGE_STRESS=1 cargo test -p azure_data_cosmos_driver \
--features fault_injection,__internal_in_memory_emulator \
--test in_memory_emulator metadata_hedging_stress -- --ignored --nocapture
```

The harness reports amplification per scenario; `HEDGE_STRESS_STORM_CONCURRENCY` controls the storm fan-out.

Contributor guide

Open the contributing guide

Research direction

Start with the in-memory emulator test entry point `metadata_hedging_stress` and run the provided `RUN_HEDGE_STRESS=1 cargo test` command with the listed features. Review the harness results and the existing metadata and data-plane hedging paths to decide the concurrency-budget behavior. Done means an agreed admission-control design is implemented and the stress scenarios verify bounded amplification.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.