Azure / Azure/azure-sdk-for-rust
[Cosmos] Weekly live GW2 legs: cross-partition query routing returns 404/1003 OwnerResourceNotFound on thin-client-multi-writer-ci
- Dominant language
- Rust
- Stars
- 884
- Forks
- 365
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 109
Description
### Bug Title
Weekly live Gateway 2.0 legs fail: cross-partition / pkrange query routing returns `404/1003 OwnerResourceNotFound` (`request_charge=0`) at the `-eastus2` regional endpoint of the pre-provisioned `thin-client-multi-writer-ci` account. Point operations on the same account pass.
### Crate Name
`azure_data_cosmos`
### Crate Version
`0.38.0`
### Description
cc the Azure Cosmos Rust SDK team — filing as an **environmental / infra** tracking issue (no product-code fix identified). Related to #4857 (move live tests to rotation-friendly fixed accounts).
The `rust - cosmos - weekly` pipeline (ADO def 7545) has been red on **every** Cosmos PR and on `main` since ~2026-08-05. Two GW2 legs fail:
- `Test ubuntu_stable_SessionSingleRegionGatewayV2`
- `Test ubuntu_stable_SessionMultiRegionGatewayV2`
Both are KEY auth (no AAD), target `--test gateway_v2` in `azure_data_cosmos`.
**Five failing tests — all cross-partition / feed-range / order-by QUERIES:**
- `gateway_v2_tests::gateway_v2_e2e::gateway_v2_cross_partition_query_full_container`
- `gateway_v2_cross_partition_query_via_feed_range_full`
- `gateway_v2_hpk_full_and_partial_partition_key_round_trip`
- `gateway_v2_query_honors_max_item_count_page_size`
- `order_by_continuation_matches_gateway_v1_and_v2`
**Error:**
```
404/1003 (OwnerResourceNotFound): ... An error occurred while routing the request
request_charge=0
endpoint https://thin-client-multi-writer-ci-eastus2.documents.azure.com/
```
Point-op tests on the **same** account PASS (6 passed / 5 failed). So the account is reachable and auth works — **only cross-partition query routing (pkranges resolution) fails**. `request_charge=0` indicates the request never reached a partition; it failed during routing/metadata resolution on the service side.
**Why this is environmental, not a code regression:**
- The account is **external and pre-provisioned** — endpoint/key come from secrets `thinclient-test-endpoint` / `thinclient-test-key` (and `thin-client-canary-multi-region-session-*`) in the variable group **"Test Secrets for Cosmos Live Tests - user administered"**. It is **not** created by repo `test-resources.bicep`, so there is no repo-side provisioning change that can fix it.
- Green→red timeline points at the account, not a merge:
- **7/19** build **6585211** — last fully green weekly on `main`.
- **8/2** build **6648854** — GW2 leg already RED, failing at account discovery with `503/20010 ... dns error ... failed to lookup address information` for `thin-client-multi-writer-ci.documents.azure.com`.
- **8/5** builds **6663144** (#4976), **6659377** (#4977) — RED as `404/1003 OwnerResourceNotFound` at the `-eastus2` regional endpoint.
- The DNS-failure → OwnerResourceNotFound evolution is classic account re-provisioning / regional-metadata drift on an externally-managed account.
### Steps to Reproduce
1. Run the `rust - cosmos - weekly` pipeline (ADO def 7545), or the `gateway_v2` test target against the `thin-client-multi-writer-ci` account.
2. Observe point operations pass but cross-partition queries fail with `404/1003 OwnerResourceNotFound`, `request_charge=0`, at `https://thin-client-multi-writer-ci-eastus2.documents.azure.com/`.
**Ask / hand-off:** owning team (the Azure Cosmos Rust SDK team) to verify the server-side routing/pkrange metadata for the `thin-client-multi-writer-ci` GW2 account (and its `-eastus2` regional endpoint), or repoint the weekly GW2 legs at a healthy account per #4857.
### Checklist
- [x] Follow our Code of Conduct
- [x] Checked that there isn't already an issue for this
- [x] Environmental/infra tracking issue (live-account routing drift), not a product-code minimal repro
Contributor guide
Research direction
Start with the `gateway_v2` test target and the five named cross-partition query tests, then run the `rust - cosmos - weekly` pipeline (ADO definition 7545) against the stated account. Compare the passing point operations with the failing queries and their routing errors. Done means the owning team verifies or repairs the account's server-side metadata, or repoints the weekly GW2 legs; the issue identifies no repository file to change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, rust
- Domain
- cloud, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100