Azure / Azure/azure-sdk-for-java

[cosmos] Hedging Detection API — public accessors on CosmosDiagnostics + CosmosDiagnosticsContext

Open
#49,182 1 comment 0 reactions 0 assignees View on GitHub
Client Cosmos needs-team-attention Service Attention
Dominant language
Java
Stars
2.6k
Forks
2.2k
Avg merge
2d 8h
Merged PRs (30d)
178

Description

## Summary

Add a public **Hedging Detection API** to `com.azure.cosmos.CosmosDiagnostics` and `com.azure.cosmos.CosmosDiagnosticsContext` so customers can post-hoc determine whether a successful or failed Cosmos point/feed operation went through cross-region hedging, which regions were dispatched against, and which regions responded. The surface is strictly additive and non-breaking — both `CosmosDiagnostics` and `CosmosDiagnosticsContext` are `final`, so additions are non-breaking by construction.

This is part of a cross-SDK feature being implemented in parallel across .NET, Java, and Python (with a spec-only deliverable for Rust). See cross-SDK design doc *"Hedging Detection API - Technical Design - Final"* (2026-05) for the contract.

## Public API additions

```java
package com.azure.cosmos.models;

public final class RequestedRegion {
public RequestedRegion(String regionName, RequestedRegionReason reason); // null-guards on regionName
public String getRegionName();
public RequestedRegionReason getReason();
@Override public boolean equals(Object o);
@Override public int hashCode();
@Override public String toString(); // ":"
}

public enum RequestedRegionReason {
INITIAL,
OPERATION_RETRY,
TRANSPORT_RETRY, // reserved
HEDGING,
REGION_FAILOVER,
CIRCUIT_BREAKER_PROBE, // tied to upstream PRs #45197 / #45267 / #46477 / #48421
}
```

```java
package com.azure.cosmos;

public final class CosmosDiagnostics {
// existing members unchanged
public boolean isHedgingStarted();
public List getRequestedRegions(); // unmodifiable snapshot
public List getRespondedRegions(); // unmodifiable snapshot, arrival order, duplicates allowed
}

public final class CosmosDiagnosticsContext {
// existing members unchanged
public boolean isHedgingStarted(); // aggregates across children
public List getRequestedRegions(); // aggregates across children (same pattern as getContactedRegionNames)
public List getRespondedRegions(); // aggregates across children
}
```

Internal state lives on `ClientSideRequestStatistics`: `private final Object regionLock = new Object();` + `List requestedRegions` + `List respondedRegions` + `boolean hedgingStarted`. **All three guarded by the same lock** — reads AND writes acquire the lock so any reader observes both writes or neither. Bridge accessor `ImplementationBridgeHelpers.CosmosDiagnosticsAccessor` gains a **single** `appendRequestedRegion(diag, entry)` method whose implementation writes both the entry and (if `reason == HEDGING`) sets `hedgingStarted = true` inside `synchronized(stats.regionLock)`. `setHedgingStarted` is **not** a separate bridge surface — the compound-atomicity invariant is enforced from the bridge.

## Critical: Reactor operator order

Append the `HEDGING` entry via `.doOnSubscribe(...)` attached **inside the hedge arm's `Mono` chain**, NOT before `Mono.delaySubscription`. Operator order matters:

```java
// CORRECT — doOnSubscribe is on the upstream of delaySubscription
Mono hedgeArm = regionalMono
.doOnSubscribe(s -> bridgeAccessor.appendRequestedRegion(
diag, new RequestedRegion(region, RequestedRegionReason.HEDGING)))
.delaySubscription(threshold, scheduler);

// WRONG — would fire before the delay starts (phantom entry on primary-wins-under-threshold)
Mono hedgeArm = regionalMono
.delaySubscription(threshold, scheduler)
.doOnSubscribe(s -> bridgeAccessor.appendRequestedRegion(...));
```

This is the design doc §12 "no phantom entries" contract — see AC2 / AC10.

## Acceptance criteria (testable)

- [ ] **AC1** Single-region client, readItem success, no retries → `isHedgingStarted() == false`; `getRequestedRegions().size() == 1` with reason `INITIAL`; `getRespondedRegions().size() == 1`.
- [ ] **AC2** Multi-region client, hedging enabled (`ThresholdBasedAvailabilityStrategy`), primary responds under threshold → `isHedgingStarted() == false`; `getRequestedRegions().size() == 1`; **no phantom `HEDGING` entry**.
- [ ] **AC3** Multi-region client, hedging enabled, primary slow, hedge arm wins → `isHedgingStarted() == true`; `getRequestedRegions()` has ≥2 entries including `(hedgeRegion, HEDGING)`; `getRespondedRegions()` has ≥1 entry.
- [ ] **AC4** 410 Gone retry on same region → `getRequestedRegions()` includes consecutive entries `(region, INITIAL)` then `(region, OPERATION_RETRY)`.
- [ ] **AC5** Region failover after endpoint failure → `getRequestedRegions()` includes `(originalRegion, INITIAL)` then `(secondaryRegion, REGION_FAILOVER)`.
- [ ] **AC6** All-regions-down error → `CosmosException.getDiagnostics().getRequestedRegions()` non-empty.
- [ ] **AC9** Existing `EndToEndTimeOutWithAvailabilityTest.getContactedRegionNames().size() > 1` (`:117`) and all 33+ assertions in `FaultInjectionWithAvailabilityStrategyTests.java` continue to pass.
- [ ] **AC10** Reactor scheduler / cancellation smoke test — assert (a) `.doOnSubscribe()` on each hedge arm fires **only after** `delaySubscription` elapses without cancellation; (b) when the primary completes before `delaySubscription` elapses, the hedge arm's `Mono` is cancelled and `.doOnSubscribe()` does NOT fire; (c) when `.doOnSubscribe()` does fire, the append happens on the orchestrator-side scheduler.
- [ ] **AC11** Aggregation — `CosmosDiagnosticsContext.getRequestedRegions()` returns the FIFO-ordered union across the `ConcurrentLinkedDeque` children.
- [ ] **AC12** Spark connector regression — `azure-cosmos-spark_3-4_2-12` test suite passes locally.
- [ ] **AC13** APIView snapshot diff cleanly shows the additions (no removals; no signature drift).
- [ ] **AC14** Live multi-region smoke test (≥1) — runs against the team's multi-region test account with hedging enabled (`ThresholdBasedAvailabilityStrategy`), injects primary-slow latency via fault-injection, and asserts `isHedgingStarted() == true`, `getRequestedRegions()` includes both regions with the secondary tagged `HEDGING`, `getRespondedRegions()` includes the secondary region.

## Files in scope

- New: `sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/models/RequestedRegion.java`, `RequestedRegionReason.java`
- Modify: `CosmosDiagnostics.java`, `CosmosDiagnosticsContext.java`, `ClientSideRequestStatistics.java` (state + `@JsonIgnore` on new fields — Spark / encryption / Kafka-connect deserialize through this class), `ImplementationBridgeHelpers.java` (bridge accessor extension), `RxDocumentClientImpl.wrapPointOperationWithAvailabilityStrategy:5460` + `executeFeedOperationWithAvailabilityStrategy:5850`, `ClientRetryPolicy.refreshLocation` / `shouldRetryOnEndpointFailureAsync` (`:245`, `:298`), `GoneAndRetryWithRetryPolicy.java:33`/`:90`, PPCB hook (rebase on PRs #45197 / #45267 / #46477 / #48421), `sdk/cosmos/azure-cosmos/CHANGELOG.md`
- Tests: new `RequestedRegionTests.java`, new `HedgingDetectionTests.java`, live-account multi-region test

## Out of scope

- Wiring into OpenTelemetry — separate work item.
- Spark connector behavioral changes — only the `@JsonIgnore` defensive annotation lands here.
- Cross-SDK companion implementations (.NET, Python, Rust) — tracked in companion issues.

## Notes for the implementer

- Full internal spec, landscape research, plan, risk register (`side-effects.json`), and questions+answers are available from the workflow author (`@NaluTripician`) on request — they are team-only and not linked here.
- Phase 1 review gate completed on 2026-05-14. Java-specific resolved decisions: shared `regionLock` pattern (matches .NET / Python); single bridge accessor `appendRequestedRegion` (no separate `setHedgingStarted`); `com.azure.cosmos.models` placement for new public types (matches `CosmosItemResponse` / `FeedResponse` precedent); at least one live multi-region smoke test required.
- This issue is being dispatched to the **Coding Agent Harness** for end-to-end implementation; reviewers may receive a draft PR shortly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.