microsoft / microsoft/mcp

[EPIC] Live-Test Reliability and Environment Stabilization

Open
#3,033 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

epic Release-KnownIssues
Dominant language
C#
Stars
3.7k
Forks
624
Avg merge
2d 20h
Merged PRs (30d)
220

Description

## Problem statement

Azure MCP live validation currently mixes product defects, shared-framework regressions, test-data failures, environment drift, flakiness, and high infrastructure cost. Individual CI incidents expose these problems, but resolving one run does not establish a reliable validation system or prevent the same failure classes from returning.

## Vision

Live-test results are actionable and repeatable. Failures clearly distinguish product defects from test or environment problems, shared regressions are detected across toolsets, fixtures remain healthy, and the cost of live validation is intentional and measurable.

## Current work

- [ ] #3029, fix Marketplace enum deserialization failures
- [x] #3030, correct shared validation-message rendering
- [ ] #3031, stabilize AKS parameter-validation and graceful-handling tests
- [ ] #3032, provide deterministic Cosmos test data
- [ ] #1949, stabilize App Configuration account discovery
- [ ] #520, eliminate Kusto live-test flakiness
- [ ] #151, reduce or justify Kusto live-test cost

## Goals (in scope)

- Classify live-test failures as product, framework, fixture, environment, or infrastructure failures
- Detect cross-tool regressions caused by shared command and validation behavior
- Make test data and deployed fixtures deterministic and diagnosable
- Reduce flaky, redundant, and unnecessarily expensive live coverage
- Preserve useful failure artifacts and ownership signals from CI runs
- Define when a scenario requires live validation instead of recorded playback

## Non-goals (out of scope)

- Completing the recorded-test migration, which is tracked by the recorded-test epic
- Treating every isolated product defect as a permanent child of this epic
- Masking failures through retries or relaxed assertions without identifying the cause

## Success criteria

- [ ] Live-test failures identify the responsible product, framework, fixture, or environment layer
- [ ] Shared regressions are detected consistently across affected toolsets
- [ ] Known flaky and environment-dependent suites have deterministic setup and cleanup
- [ ] Live-test cost and execution time have documented baselines
- [ ] Resolved failures leave regression coverage or an explicit accepted-risk decision

## Dependencies

- Shared live-test framework and Azure test-resource infrastructure
- Recorded-test guidance for scenarios that do not require live validation
- CI diagnostics, retained artifacts, and service-team ownership

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the listed child issues (#3029–#3032, #1949, #520, and #151) and the shared live-test framework and Azure test-resource infrastructure named in the issue. Map the failure classes, fixture concerns, diagnostics, and cost baselines before choosing a focused child task. Done means the relevant success criterion is met with regression coverage or an explicit accepted-risk decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
cloud, devops, infrastructure, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.