microsoft / microsoft/GitHub-Copilot-for-Azure

[Feature]: probation integration tests

Open
#2,966 1 comment 0 reactions 0 assignees View on GitHub
assign-to-copilot integration-test skills
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

Unlike determinstic software, model "upgrades" don't always improve things for scenarios we care about. They can even introduce bugs that cause significantly worse outcomes (e.g. Gemini infinite loop bug). We may seek a way to retain tests for recently retired skills and see if they need to be re-introduced.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named in the issue. Start by locating the existing integration-test coverage and records for retired skills, then determine how model upgrades and probation would be represented. Done would mean the project has an agreed workflow for retaining, evaluating, and potentially reintroducing retired-skill tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.