Azure / Azure/azure-sdk-for-rust

[Cosmos] Weekly live GW2 legs: cross-partition query routing returns 404/1003 OwnerResourceNotFound on thin-client-multi-writer-ci

Open
#4,996 0 comments 1 reaction 0 assignees View on GitHub
Cosmos test
Dominant language
Rust
Stars
884
Forks
365
Avg merge
2d 19h
Merged PRs (30d)
109

Description

### Bug Title

Weekly live Gateway 2.0 legs fail: cross-partition / pkrange query routing returns `404/1003 OwnerResourceNotFound` (`request_charge=0`) at the `-eastus2` regional endpoint of the pre-provisioned `thin-client-multi-writer-ci` account. Point operations on the same account pass.

### Crate Name

`azure_data_cosmos`

### Crate Version

`0.38.0`

### Description

cc the Azure Cosmos Rust SDK team — filing as an **environmental / infra** tracking issue (no product-code fix identified). Related to #4857 (move live tests to rotation-friendly fixed accounts).

The `rust - cosmos - weekly` pipeline (ADO def 7545) has been red on **every** Cosmos PR and on `main` since ~2026-08-05. Two GW2 legs fail:

- `Test ubuntu_stable_SessionSingleRegionGatewayV2`
- `Test ubuntu_stable_SessionMultiRegionGatewayV2`

Both are KEY auth (no AAD), target `--test gateway_v2` in `azure_data_cosmos`.

**Five failing tests — all cross-partition / feed-range / order-by QUERIES:**

- `gateway_v2_tests::gateway_v2_e2e::gateway_v2_cross_partition_query_full_container`
- `gateway_v2_cross_partition_query_via_feed_range_full`
- `gateway_v2_hpk_full_and_partial_partition_key_round_trip`
- `gateway_v2_query_honors_max_item_count_page_size`
- `order_by_continuation_matches_gateway_v1_and_v2`

**Error:**

```
404/1003 (OwnerResourceNotFound): ... An error occurred while routing the request
request_charge=0
endpoint https://thin-client-multi-writer-ci-eastus2.documents.azure.com/
```

Point-op tests on the **same** account PASS (6 passed / 5 failed). So the account is reachable and auth works — **only cross-partition query routing (pkranges resolution) fails**. `request_charge=0` indicates the request never reached a partition; it failed during routing/metadata resolution on the service side.

**Why this is environmental, not a code regression:**

- The account is **external and pre-provisioned** — endpoint/key come from secrets `thinclient-test-endpoint` / `thinclient-test-key` (and `thin-client-canary-multi-region-session-*`) in the variable group **"Test Secrets for Cosmos Live Tests - user administered"**. It is **not** created by repo `test-resources.bicep`, so there is no repo-side provisioning change that can fix it.
- Green→red timeline points at the account, not a merge:
- **7/19** build **6585211** — last fully green weekly on `main`.
- **8/2** build **6648854** — GW2 leg already RED, failing at account discovery with `503/20010 ... dns error ... failed to lookup address information` for `thin-client-multi-writer-ci.documents.azure.com`.
- **8/5** builds **6663144** (#4976), **6659377** (#4977) — RED as `404/1003 OwnerResourceNotFound` at the `-eastus2` regional endpoint.
- The DNS-failure → OwnerResourceNotFound evolution is classic account re-provisioning / regional-metadata drift on an externally-managed account.

### Steps to Reproduce

1. Run the `rust - cosmos - weekly` pipeline (ADO def 7545), or the `gateway_v2` test target against the `thin-client-multi-writer-ci` account.
2. Observe point operations pass but cross-partition queries fail with `404/1003 OwnerResourceNotFound`, `request_charge=0`, at `https://thin-client-multi-writer-ci-eastus2.documents.azure.com/`.

**Ask / hand-off:** owning team (the Azure Cosmos Rust SDK team) to verify the server-side routing/pkrange metadata for the `thin-client-multi-writer-ci` GW2 account (and its `-eastus2` regional endpoint), or repoint the weekly GW2 legs at a healthy account per #4857.

### Checklist

- [x] Follow our Code of Conduct
- [x] Checked that there isn't already an issue for this
- [x] Environmental/infra tracking issue (live-account routing drift), not a product-code minimal repro

Contributor guide

Open the contributing guide

Research direction

Start with the `gateway_v2` test target and the five named cross-partition query tests, then run the `rust - cosmos - weekly` pipeline (ADO definition 7545) against the stated account. Compare the passing point operations with the failing queries and their routing errors. Done means the owning team verifies or repairs the account's server-side metadata, or repoints the weekly GW2 legs; the issue identifies no repository file to change.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, rust
Domain
cloud, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.