microsoft / microsoft/GitHub-Copilot-for-Azure

Compute a per skill "coverage" list for integration tests

Open
#1,510 1 comment 0 reactions 0 assignees View on GitHub
integration-test skills untriaged
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

Good test coverage indicates the tests cover all possible ways the test subject can be involved. For traditional software, it means every line of code that can be executed. For skills, it means our test prompts cover enough scenarios where the skill is expected to help the agent. We can aggregate the test prompts and show a per skill list of all the test prompts being used.

Display this per skill list of prompts in the dashboard allows us to review the covered scenarios for each skill and ask the following question: "Is there another prompt that should have resulted in using the skill to accomplish something that hasn't been covered by existing tests?" This is assuming each test case runs one prompt. This happens to be the case for all existing integration tests and there is no foreseeable plan to change that.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the existing integration-test prompt aggregation and dashboard skill views; verify that each test case contributes one prompt, then confirm the dashboard shows a per-skill prompt list.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
analytics, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.