[Cosmos] Weekly live MultiWrite: West US 3 satellite credential drift (KEY master-key + AAD issuer; #4896 exonerated)

### Bug Title

Weekly live MultiWrite legs fail on failover to the **West US 3 satellite** of the freshly-provisioned multi-write account, on BOTH auth modes: the KEY leg gets `401 The MAC signature ... is not the same as the computed signature`, and the AAD leg gets `403/5302 RbacUnauthorizedNameBasedDataRequest` evolving to `401/5007 AadTokenInvalidIssuer [sts.windows.net/...]`. The **hub (East US 2) accepts** the same key/token; only the **satellite (West US 3)** rejects correctly-credentialed requests. Both are facets of the same satellite-region **credential drift** — **environmental, not a code regression** (commit `3c97d9daf9` / #4896 is exonerated, see below). Related to #4857.

### Crate Name

`azure_data_cosmos` (driver: `azure_data_cosmos_driver` 0.7.0)

### Crate Version

`0.38.0`

### Description

cc the Azure Cosmos Rust SDK team and Nalu Tripician (author of #4896) — filing as a single **environmental / satellite-region credential-config** tracking issue covering both the AAD-issuer and KEY-master-key facets, and to explicitly **clear #4896**. Consolidates the previously-separate KEY issue (#4998, now closed into this one) per maintainer request.

The `rust - cosmos - weekly` pipeline (ADO def 7545) legs `Test ubuntu_stable_SessionMultiWrite` (KEY) and `Test ubuntu_stable_aad_auth_SessionMultiWrite` (AAD) — both `--test multi_write` in `azure_data_cosmos` — fail on the cross-region fault-injection / retry tests that fail over to the **West US 3 satellite**:

- `cosmos_multi_write_fault_injection::{fault_injection_read_region_retry_503, fault_injection_write_connection_error_failover, fault_injection_transport_generated_503_write_retries_via_failover, fault_injection_connection_error_local_retry_succeeds}`
- `cosmos_multi_write_retry_policies::{query_cross_region_retry_on_408, read_cross_region_retry_on_408, read_cross_region_retry_on_500}`

---

## Facet 1 — AAD: satellite rejects a globally-scoped token (`401/5007 AadTokenInvalidIssuer`)

**Error (on the request hedged/failed over to the satellite):**

```
401/5007 (AadTokenInvalidIssuer): Provided AAD token was issued by the authority [https://sts.windows.net/...]
```
On 8/2 this surfaced instead as `403/5302 RbacUnauthorizedNameBasedDataRequest` on the hub, evolving into the satellite `AadTokenInvalidIssuer` by 8/5.

**Why environmental, not code:**
- The driver requests AAD tokens for a **single global scope** `https://cosmos.azure.com/.default` (`COSMOS_AAD_SCOPE` in `authorization_policy.rs`) — **not** region-scoped. The identical token is sent to hub and satellite.
- The **hub accepts** the token; **only the satellite rejects** it with `AadTokenInvalidIssuer`. A satellite-only issuer rejection for a globally-scoped token is a **server-side / tenant credential-trust** problem on that region, not something the client selects.
- `AadTokenInvalidIssuer [sts.windows.net/...]` is a classic issuer/tenant-authority mismatch — consistent with the ephemeral test-tenant rotation described in #4857.

---

## Facet 2 — KEY: satellite rejects a correctly-signed request (`401 MAC signature mismatch`)

**Error (on the `region_failover` write to the satellite):**

```
401: The MAC signature found in the HTTP request is not the same as the computed signature.
```
context: "write should succeed after failover to satellite", endpoint `...-westus3.documents.azure.com`. The logged string-to-sign is correct/name-based (e.g. `get\ndocs\ndbs/.../colls/.../docs/...\n\n\n`).

### Determination: satellite-region master-key convergence, NOT a code regression

Initial suspicion fell on #4896 (`3c97d9daf9`) because the green→red flip straddled its merge. A hunk-by-hunk diff review **refutes** that. Dispositive evidence:

1. **Two of the seven failing tests are WRITE failovers that never hedge.** `fault_injection_transport_generated_503_write_retries_via_failover` and `fault_injection_write_connection_error_failover` fail over via the **main-loop** retry path (`execution_context = region_failover`), not the hedging path #4896 added — and they still 401 on westus3. A bug in the new hedging code cannot explain failures on a path hedging never runs.
2. **The signed request is built/signed by code #4896 left byte-for-byte identical** (`build_transport_request`, `operation_pipeline.rs:1723`, and the transport sign/send blocks — verified by diffing `3c97d9daf9^` vs `3c97d9daf9`).
3. **The entire #4896 `operation_pipeline.rs` diff only touches hedge orchestration**, never request build/sign. `routing_decision_for_pinned_endpoint` changes only the target URL/host — which is **not part of the Cosmos string-to-sign** — and only fires when `region_pin` is set, never these data-plane document tests.
4. **Signing is provably host-independent.** `build_string_to_sign` (`authorization_policy.rs:148`) signs `verb\nresource_type\nresource_link\ndate\n\n` — the host is never included. `sign_request` (`request_signing.rs:25`) stamps `x-ms-date` and `Authorization` atomically from the **same** freshly generated date on every attempt. None of these files were touched by any 8/4 merge.
5. **Same signing code + same key succeed 23x in the same run.** On 8/5 all hub-region tests pass using the identical signing code and identical account-global master key; only satellite-bound requests 401.

**Positive environmental mechanism.** The multi-write account is provisioned fresh each run (repo bicep: East US 2 hub + West US 3 satellite; master key is account-global). The write-failover test asserts a **successful** write on the satellite and **passed on 8/2** (build **6648854**: multi_write binary `test result: ok. 30 passed; 0 failed`). By 8/5 the same seven fail with the MAC mismatch. A 401 MAC mismatch for a correctly and identically signed request means the **satellite validated against a different/stale master key** than the account-global key the client used — i.e. West US 3's key metadata had not converged for the freshly-provisioned account.

### Regression guard added (empirical exoneration)

Two guard tests were added to `operation_pipeline.rs` (tests module) and **pass on current `main` (with #4896 merged)**:

- `failover_to_satellite_signs_over_identical_content_and_binds_date` (`operation_pipeline.rs:4397`) — a point read routed to hub vs satellite targets **different hosts** but produces an **identical resource path, identical `auth_context`, and identical signature** for a given date, and the signature **binds the date**.
- `satellite_attempt_authorization_matches_its_own_x_ms_date` (`operation_pipeline.rs:4497`) — drives the real `sign_request` on a satellite-bound request and asserts the emitted `Authorization` equals the signature recomputed over the request's own `x-ms-date`.

```
running 2 tests
test ...::satellite_attempt_authorization_matches_its_own_x_ms_date ... ok
test ...::failover_to_satellite_signs_over_identical_content_and_binds_date ... ok
test result: ok. 2 passed; 0 failed
```

These passing on #4896-inclusive code is direct evidence the failover/hedge signing path is invariant across the endpoint swap. Handed off in draft PR #4999.

---

## Shared timeline (both facets)

- **7/19** build **6585211** — last fully green weekly on `main`.
- **8/2** build **6648854** — KEY `SessionMultiWrite` leg GREEN (`30 passed; 0 failed`); AAD leg already RED (`403/5302 RbacUnauthorizedNameBasedDataRequest`).
- **8/5** builds **6663144** (#4976), **6659377** (#4977) — KEY leg RED with satellite MAC mismatch (7 multi-attempt tests); AAD leg RED with `401/5007 AadTokenInvalidIssuer [sts.windows.net/...]`.

The KEY leg passing on 8/2 and failing on 8/5 with **no relevant main merges in between** is the temporal proof that both facets are drift of the fresh multi-write account's **satellite-region credentials**, not code.

### Steps to Reproduce

1. Run the `rust - cosmos - weekly` pipeline (ADO def 7545), legs `Test ubuntu_stable_SessionMultiWrite` (KEY) and `Test ubuntu_stable_aad_auth_SessionMultiWrite` (AAD).
2. Observe the fault-injection/retry tests fail over to the West US 3 satellite and receive `401 MAC signature mismatch` (KEY) / `401/5007 AadTokenInvalidIssuer` (AAD), while all hub-region requests succeed with the same key/token.

**Ask / hand-off:** owning team (the Azure Cosmos Rust SDK team) to verify, for the freshly-provisioned multi-write account's **West US 3** region: (a) the master-key metadata converges before the failover tests run (region key-replication / provisioning readiness), and (b) the AAD app-registration / issuer trust matches the CI credential's issuing authority — or repoint both legs at a rotation-friendly account per #4857. **No product-code fix is warranted; #4896 (`3c97d9daf9`) is not the cause.** A repo-side resilience change (satellite-region readiness warmup + fail-loud infra-drift signal) is in PR #5000.

### Checklist

- [x] Follow our Code of Conduct
- [x] Checked that there isn't already an issue for this
- [x] Environmental/credential-config tracking issue (satellite-region key convergence + AAD issuer trust); #4896 exonerated with a passing regression guard

Start with ADO definition 7545 and the two SessionMultiWrite legs, then read the listed failover tests in azure_data_cosmos. Review operation_pipeline.rs, authorization_policy.rs, and request_signing.rs alongside the two regression guards. Done means the owning team verifies West US 3 key convergence and AAD issuer trust, or provides an agreed infrastructure-readiness change; the issue states that no product-code fix is warranted.

Written by the indexing model from the issue text.