microsoft / microsoft/GitHub-Copilot-for-Azure
[Epic] Detect workflow regressions before users encounter them
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Problem statement
Build and structure checks do not prove that skills route correctly, choose appropriate tools, preserve context, and complete real Azure workflows. Regressions are often discovered only through production or dogfooding.
## Outcome
Representative workflow regressions are detected with actionable evidence before release.
## Goals
- Expand routing, negative, integration, and end-to-end evaluations
- Maintain representative datasets and baselines
- Test supported models and clients where behavior differs
- Produce actionable failure evidence and ownership
## Non-goals
- Guaranteeing deterministic model behavior
- Treating every historical failure as an active feature
## Success criteria
- [ ] Priority workflows have positive and negative coverage
- [ ] Release gates detect material regressions
- [ ] Failures identify the affected skill, tool, client, and scenario
## Dependencies
Vally, integration runners, model access, client test surfaces, telemetry, and scenario owners.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by mapping the priority workflows, dependencies, and evaluation surfaces listed in the issue, including Vally, integration runners, model access, client test surfaces, and telemetry. Done means representative positive and negative coverage, release gates for material regressions, and failure evidence identifying the affected skill, tool, client, and scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- ai, cloud, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100