[EPIC] Live-Test Reliability and Environment Stabilization
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Problem statement
Azure MCP live validation currently mixes product defects, shared-framework regressions, test-data failures, environment drift, flakiness, and high infrastructure cost. Individual CI incidents expose these problems, but resolving one run does not establish a reliable validation system or prevent the same failure classes from returning.
## Vision
Live-test results are actionable and repeatable. Failures clearly distinguish product defects from test or environment problems, shared regressions are detected across toolsets, fixtures remain healthy, and the cost of live validation is intentional and measurable.
## Current work
- [ ] #3029, fix Marketplace enum deserialization failures
- [x] #3030, correct shared validation-message rendering
- [ ] #3031, stabilize AKS parameter-validation and graceful-handling tests
- [ ] #3032, provide deterministic Cosmos test data
- [ ] #1949, stabilize App Configuration account discovery
- [ ] #520, eliminate Kusto live-test flakiness
- [ ] #151, reduce or justify Kusto live-test cost
## Goals (in scope)
- Classify live-test failures as product, framework, fixture, environment, or infrastructure failures
- Detect cross-tool regressions caused by shared command and validation behavior
- Make test data and deployed fixtures deterministic and diagnosable
- Reduce flaky, redundant, and unnecessarily expensive live coverage
- Preserve useful failure artifacts and ownership signals from CI runs
- Define when a scenario requires live validation instead of recorded playback
## Non-goals (out of scope)
- Completing the recorded-test migration, which is tracked by the recorded-test epic
- Treating every isolated product defect as a permanent child of this epic
- Masking failures through retries or relaxed assertions without identifying the cause
## Success criteria
- [ ] Live-test failures identify the responsible product, framework, fixture, or environment layer
- [ ] Shared regressions are detected consistently across affected toolsets
- [ ] Known flaky and environment-dependent suites have deterministic setup and cleanup
- [ ] Live-test cost and execution time have documented baselines
- [ ] Resolved failures leave regression coverage or an explicit accepted-risk decision
## Dependencies
- Shared live-test framework and Azure test-resource infrastructure
- Recorded-test guidance for scenarios that do not require live validation
- CI diagnostics, retained artifacts, and service-team ownership
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the listed child issues (#3029–#3032, #1949, #520, and #151) and the shared live-test framework and Azure test-resource infrastructure named in the issue. Map the failure classes, fixture concerns, diagnostics, and cost baselines before choosing a focused child task. Done means the relevant success criterion is met with regression coverage or an explicit accepted-risk decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- cloud, devops, infrastructure, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100