microsoft / microsoft/mcp

Make VS Code integration tests deterministic and secure

Open
#3,093 0 comments 0 reactions 0 assignees View on GitHub
bug Central-EngSys server-Azure.Mcp test issue
Dominant language
C#
Stars
3.7k
Forks
624
Avg merge
2d 20h
Merged PRs (30d)
220

Description

## Problem

The VS Code elicitation integration test added in #2549 starts Azure MCP through `npx -y @azure/mcp@latest` in two test-harness paths. Each run therefore resolves and executes a mutable published package instead of an explicitly selected server build.

This causes four problems:

- Failures are not reproducible because the server version can change between runs.
- The pipeline downloads and executes code that was not selected or reviewed with the tested commit.
- Regressions in the current commit are not detected until a new package is published.
- The harness is hardcoded to Azure MCP, which prevents straightforward reuse for other servers.

PR #3095 made the Playwright dependency deterministic and removed the unused Build-stage dependency. It intentionally did not change how the MCP server under test is selected.

## Required outcome

- Remove both `@azure/mcp@latest` launches from the outerloop harness.
- Select the server executable or package explicitly and use the same selection in both harness paths.
- Do not resolve or execute a mutable MCP server version during the test.
- Fail clearly when the selected server input is missing.
- Preserve the existing VS Code elicitation scenario and assertions.
- Make the pipeline dependencies match the inputs the test actually consumes.

## Preferred implementation

Consume the Azure MCP build artifact produced by the same pipeline run and restore an explicit dependency on the producing build job or stage. This makes the test deterministic and catches regressions in the current commit.

If this lane is intentionally retained as a published-package compatibility test instead, pin an exact `@azure/mcp` version in `package.json` and `package-lock.json`, invoke it with `npm exec --no --`, and use Dependabot for controlled updates. The lane must document which of these two contracts it validates.

Generalizing the harness to every MCP server is follow-up work, not required to remove the current nondeterministic and mutable package execution.

## Acceptance criteria

- No test path invokes `@azure/mcp@latest` or another mutable package reference.
- Both server launches use one explicit, configurable server input.
- Re-running the same commit uses the same server bits.
- The existing elicitation integration test still validates the expected consent prompt.
- Pipeline ordering and artifact or package setup are covered by the integration-test configuration.

Contributor guide

Open the contributing guide

Research direction

Start with the two outerloop harness paths that launch Azure MCP and the integration-test configuration that controls pipeline ordering and dependencies. Review the VS Code elicitation integration test and its existing consent-prompt assertions, then run it with the selected server input. Done means both paths use the same explicit, non-mutable input, fail clearly when it is missing, and consume the producing artifact or pinned package deterministically.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, node.js, playwright, vscode
Domain
ci-cd, devtools, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.