microsoft / microsoft/GitHub-Copilot-for-Azure
[Feature]: probation integration tests
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
Unlike determinstic software, model "upgrades" don't always improve things for scenarios we care about. They can even introduce bugs that cause significantly worse outcomes (e.g. Gemini infinite loop bug). We may seek a way to retain tests for recently retired skills and see if they need to be re-introduced.
Contributor guide
Research direction
No files, tests, or entry points are named in the issue. Start by locating the existing integration-test coverage and records for retired skills, then determine how model upgrades and probation would be represented. Done would mean the project has an agreed workflow for retaining, evaluating, and potentially reintroducing retired-skill tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100