Azure / Azure/azure-sdk-for-python
[Cosmos] Move live tests to rotation-friendly fixed accounts
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
**Is your feature request related to a problem? Please describe.**
Python Cosmos key-auth live-test matrix jobs currently deploy `sdk/cosmos/test-resources.bicep` through the shared `eng/common/TestResources` flow, export `ACCOUNT_HOST`/`ACCOUNT_KEY`, run `azpysdk whl`/pytest marker lanes, and remove the account afterward (`sdk/cosmos/tests.yml`, `eng/pipelines/templates/jobs/live.tests.yml`). This per-run central provisioning model does not fit an ephemeral tenant that rotates roughly every three months, and it makes overlapping-run safety difficult to validate independently from provisioning.
The package has a broad sync/async test surface sharing `TestConfig.TEST_DATABASE_ID`; session hooks in `tests/conftest.py` create/delete a database and common containers. Specialized pytest lanes include split, query/long, multi-region/multi-write, circuit breaker, per-partition automatic failover, AAD, semantic reranker, and GSI. Some tests already use UUIDs and owned-resource cleanup, but fixed shared accounts require a deliberate cross-run interference audit before provisioning is disabled.
Prior art: Azure/azure-sdk-for-java#49735. Python should capture its lessons without assuming an identical implementation. Fixed accounts exposed global feed/count assumptions, weak random names, broad stale cleanup, non-thread-safe async collectors, destructive retries that reused resources, unit-only lifecycle hooks that accidentally reached live/emulator endpoints, and control-plane timeouts.
**Describe the solution you'd like**
Migrate only key-based Python Cosmos live tests to fixed, team-owned accounts that can be recreated after tenant rotation. Keep overlapping runs logically isolated; do not add a pipeline-wide lease or global build serialization.
### Discovery and account inventory
- [ ] Inventory every account shape/topology/capability required by sync and async tests and map each key-auth pytest marker/matrix leg in `sdk/cosmos/live-platform-matrix.json` to a logical account selector.
- [ ] At minimum validate the current shapes implied by the matrix/Bicep: standard single-region Session, multi-write multi-region Session, single-write multi-region Session, circuit-breaker variants, per-partition automatic failover, split/throughput workloads, vector/full-text capabilities, and the separately configured GSI account.
- [ ] Determine whether semantic-reranker, integrated-cache, fault-injection, control-plane, or other specialized tests run in another pipeline/config and need dedicated accounts/selectors.
- [ ] Keep AAD/service-principal lanes (`COSMOS_TEST_DATA_AUTH_MODE=aad`) that require a different tenant or identity configuration outside this key-based migration unless their design is explicitly included.
### Fixed-account provisioning and secret resolution
- [ ] Add a checked-in declarative account definition plus idempotent provisioning/reconciliation script under an appropriate Cosmos pipeline/test-resource directory; deploy into a team-owned resource group.
- [ ] Support dry-run/what-if and reconciliation so the same definitions recreate accounts after tenant rotation without embedding subscription or tenant details.
- [ ] Store all logical-account endpoints plus primary and secondary keys in one schema-versioned JSON secret sourced from Key Vault/ADO variable group. Commit only a schema and sanitized sample.
- [ ] Add resolver unit tests and a reusable pipeline pre-step that validates the JSON/schema, selects the matrix leg's logical account, masks secrets, and exports the existing Python variables (`ACCOUNT_HOST`, `ACCOUNT_KEY`, and any explicitly introduced secondary-key variable).
- [ ] Preserve matrix-controlled non-secret settings such as auth mode and account behavior flags rather than duplicating them in the secret.
- [ ] Disable the per-run deploy/remove path only after all fixed accounts and the versioned secret exist and the resolver succeeds. The exact Python template hook/bypass is an implementation decision; today `tests.yml` extends `archetype-sdk-tests.yml`, which invokes `live.tests.yml` deployment before `BeforeTestSteps`.
### Concurrency hardening
- [ ] Make database, container, lease/processor, item, and other mutable resource names run-unique, preferably using UUID plus pipeline/run identity. Never rely on short/random-only names for isolation.
- [ ] Track ownership and delete only resources created by the current test/run. Remove broad stale-database/container cleanup from suites that may overlap.
- [ ] Replace account-global exact-count/feed assertions with semantic containment/ID assertions or queries scoped to the run-owned database/container.
- [ ] Audit `tests/conftest.py`, `tests/test_config.py`, class/function fixtures, cleanup finalizers, sync/async parametrization, retries, and client lifecycle. Ensure unit-only marker selections cannot accidentally connect to a fixed live account.
- [ ] Make async result collection and shared mutable state concurrency-safe; close async clients deterministically.
- [ ] Retry only bounded, known-transient operations. Do not retry destructive lifecycle sequences against a partially reused database/container.
- [ ] Isolate irreversible/heavy workloads with dedicated selectors/accounts where required: partition splits/throughput changes, change feed/processor leases, fault injection, control-plane operations, multi-region/session/consistency profiles, circuit breaker/failover, GSI, and other capability-gated tests.
- [ ] Diagnose logical name/state conflicts separately from account-level 429/capacity contention; account sizing or matrix throttling must not hide isolation defects.
### Validation, recordings, and operations
- [ ] Add targeted key-auth sync and async live validation for each logical account selector, followed by overlapping matrix runs without a global lease.
- [ ] Document recording/playback implications. The live job currently sets `AZURE_RUN_MODE=Live` and `AZURE_SKIP_LIVE_RECORDING=true`, and the package has no checked-in `tests/recordings` tree; ensure resolver values never enter recordings/logs and playback/unit selections remain network-isolated.
- [ ] Add CI acceptance coverage for missing/malformed secret JSON, unknown selector, missing endpoint/key fields, masking/export behavior, provisioning dry-run/reconciliation, and failure before pytest when resolution is invalid.
- [ ] Add a tenant-rotation runbook covering account recreation, identity/RBAC prerequisites, secret regeneration/versioning, smoke validation, rollback, and ownership/on-call expectations.
- [ ] Prove per-run account creation and teardown are no longer used by migrated key-auth lanes, while unaffected AAD/emulator/specialized lanes retain their intended provisioning path.
- [ ] Prove overlapping runs do not delete or observe each other's resources and do not require global pipeline serialization.
No secrets, keys, account endpoints, or concrete subscription/tenant IDs belong in this issue or the repository.
**Describe alternatives you've considered**
1. Keep centrally provisioned accounts per run. This preserves isolation but does not meet rotation/self-ownership goals and retains the central provisioning dependency.
2. Serialize the entire Cosmos live pipeline or acquire a pipeline-wide lease. This reduces collisions but lowers throughput, risks abandoned leases, and masks unsafe shared-account assumptions.
3. Use one fixed account with broad stale-resource cleanup. This is unsafe for overlapping runs because one run can delete or count another run's resources.
4. Move AAD and key-auth tests together. This couples unrelated tenant/identity requirements; key-auth migration should land independently unless AAD account ownership is explicitly designed.
**Additional context**
Verified Python-specific surfaces:
- `sdk/cosmos/tests.yml`: live-test matrix, max parallelism 8, user-administered variable group, GSI secret mapping.
- `sdk/cosmos/live-platform-matrix.json`: pytest marker legs and ARM topology parameters.
- `sdk/cosmos/test-resources.bicep`: current account, regions, capabilities, role assignment, and `ACCOUNT_HOST`/`ACCOUNT_KEY` outputs.
- `sdk/cosmos/azure-cosmos/pytest.ini`: Cosmos live marker taxonomy.
- `sdk/cosmos/azure-cosmos/tests/conftest.py` and `tests/test_config.py`: session-wide database/container lifecycle, key/AAD client selection, UUID-based shared IDs, sync/async helpers, and split control-plane operations.
- `test_change_feed.py`/`test_change_feed_async.py`, split tests, and GSI tests show representative owned-resource, empty-feed/count, control-plane, and specialized-account behaviors that need auditing.
Unknowns to resolve during design: the complete set of live pipelines outside `sdk/cosmos/tests.yml`; whether GSI's existing dedicated secret should be folded into the versioned account map; secondary-key consumers; required account capacity/quotas under overlap; and which heavy tests need exclusive logical accounts rather than merely unique database/container names.
Contributor guide
Assessment
This issue has not been assessed yet.