microsoft / microsoft/GitHub-Copilot-for-Azure

[Epic] Detect workflow regressions before users encounter them

Open
#2,917 1 comment 0 reactions 0 assignees View on GitHub
epic
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Problem statement

Build and structure checks do not prove that skills route correctly, choose appropriate tools, preserve context, and complete real Azure workflows. Regressions are often discovered only through production or dogfooding.

## Outcome

Representative workflow regressions are detected with actionable evidence before release.

## Goals

- Expand routing, negative, integration, and end-to-end evaluations
- Maintain representative datasets and baselines
- Test supported models and clients where behavior differs
- Produce actionable failure evidence and ownership

## Non-goals

- Guaranteeing deterministic model behavior
- Treating every historical failure as an active feature

## Success criteria

- [ ] Priority workflows have positive and negative coverage
- [ ] Release gates detect material regressions
- [ ] Failures identify the affected skill, tool, client, and scenario

## Dependencies

Vally, integration runners, model access, client test surfaces, telemetry, and scenario owners.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by mapping the priority workflows, dependencies, and evaluation surfaces listed in the issue, including Vally, integration runners, model access, client test surfaces, and telemetry. Done means representative positive and negative coverage, release gates for material regressions, and failure evidence identifying the affected skill, tool, client, and scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
ai, cloud, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.