[EPIC] Complete Recorded-Test Migration
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Problem statement
The original recorded-test migration tracker, #1208, was closed after most Azure MCP toolsets moved from live-only validation to deterministic playback. Several suites still depend on live infrastructure, and Fabric live testing needs recording support from its initial implementation. This leaves inconsistent validation behavior, slower feedback, avoidable Azure resource dependencies, and gaps in pull request coverage.
## Vision
All Azure and Fabric MCP service tests use the shared recorded-test infrastructure where practical. Pull requests run deterministic playback by default, while intentional live runs produce maintainable recordings through a consistent workflow.
## Current open work
- [ ] #1300, migrate Speech live tests to recorded tests
- [ ] #1294, migrate PostgreSQL live tests to recorded tests
- [ ] #1281, migrate Confidential Ledger live tests to recorded tests
- [ ] #2514, add Fabric live testing with recording support from inception
## Completed foundation
- #1208 established the original migration effort
- #1390 changed CI to run recorded tests by default
- #1887 and #2718 expanded recorded-test CI coverage and dependency triggers
- #1273, #1279, #1280, #1282, #1283, and #1286 completed earlier toolset migrations
- `docs/recorded-tests.md` documents the required recording and playback model
## Goals (in scope)
- Complete migration of the remaining live-only Azure MCP suites
- Ensure new Fabric live tests support recording and playback
- Use the shared test proxy, sanitization, assets, and recording conventions
- Run deterministic playback in normal pull request validation
- Preserve an intentional live-test path for creating and refreshing recordings
- Remove obsolete live-only fixtures and infrastructure after each migration
## Non-goals (out of scope)
- Eliminating live validation where Azure behavior cannot be represented safely in recordings
- Rewriting unrelated command or service implementations
- Treating general CI flakiness as part of this migration unless caused by recorded-test infrastructure
- Reopening the completed historical tracker solely to hold new work
## Success criteria
- [ ] Every remaining target suite has deterministic playback coverage
- [ ] Playback runs without Azure credentials or deployed test resources
- [ ] Record mode can create sanitized recordings using the documented workflow
- [ ] CI executes the migrated suites on relevant code and dependency changes
- [ ] Live-only fixtures, secrets, and infrastructure are removed where no longer required
- [ ] The historical tracker and completed migrations are linked for traceability
## Dependencies
- Shared Azure SDK test proxy and assets repository
- Stable sanitization and recording conventions in `docs/recorded-tests.md`
- Toolset-specific test resources for intentional record-mode runs
- Fabric test infrastructure capable of supporting the same deterministic workflow
Contributor guide
Research direction
Read docs/recorded-tests.md first, then choose one of the remaining child issues: #1300, #1294, #1281, or #2514. Compare its test resources with the shared proxy, sanitization, assets, and recording conventions; done means deterministic playback, a documented record path, relevant CI coverage, and removal of obsolete live-only infrastructure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- ci-cd, cloud, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100