Make VS Code integration tests deterministic and secure
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Problem
The VS Code elicitation integration test added in #2549 starts Azure MCP through `npx -y @azure/mcp@latest` in two test-harness paths. Each run therefore resolves and executes a mutable published package instead of an explicitly selected server build.
This causes four problems:
- Failures are not reproducible because the server version can change between runs.
- The pipeline downloads and executes code that was not selected or reviewed with the tested commit.
- Regressions in the current commit are not detected until a new package is published.
- The harness is hardcoded to Azure MCP, which prevents straightforward reuse for other servers.
PR #3095 made the Playwright dependency deterministic and removed the unused Build-stage dependency. It intentionally did not change how the MCP server under test is selected.
## Required outcome
- Remove both `@azure/mcp@latest` launches from the outerloop harness.
- Select the server executable or package explicitly and use the same selection in both harness paths.
- Do not resolve or execute a mutable MCP server version during the test.
- Fail clearly when the selected server input is missing.
- Preserve the existing VS Code elicitation scenario and assertions.
- Make the pipeline dependencies match the inputs the test actually consumes.
## Preferred implementation
Consume the Azure MCP build artifact produced by the same pipeline run and restore an explicit dependency on the producing build job or stage. This makes the test deterministic and catches regressions in the current commit.
If this lane is intentionally retained as a published-package compatibility test instead, pin an exact `@azure/mcp` version in `package.json` and `package-lock.json`, invoke it with `npm exec --no --`, and use Dependabot for controlled updates. The lane must document which of these two contracts it validates.
Generalizing the harness to every MCP server is follow-up work, not required to remove the current nondeterministic and mutable package execution.
## Acceptance criteria
- No test path invokes `@azure/mcp@latest` or another mutable package reference.
- Both server launches use one explicit, configurable server input.
- Re-running the same commit uses the same server bits.
- The existing elicitation integration test still validates the expected consent prompt.
- Pipeline ordering and artifact or package setup are covered by the integration-test configuration.
Contributor guide
Research direction
Start with the two outerloop harness paths that launch Azure MCP and the integration-test configuration that controls pipeline ordering and dependencies. Review the VS Code elicitation integration test and its existing consent-prompt assertions, then run it with the selected server input. Done means both paths use the same explicit, non-mutable input, fail clearly when it is missing, and consume the producing artifact or pinned package deterministically.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, node.js, playwright, vscode
- Domain
- ci-cd, devtools, testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100