microsoft / microsoft/GitHub-Copilot-for-Azure
[Epic] Deterministic integration tests with Azure fixture
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Problem
Some skills are designed to operate on existing Azure resources, such as azure-resource-lookup, azure-resource-visualizer, azure-diagnostics. As of today, we don't have a well defined solution for providing Azure fixtures to integration tests to evaluate how to affect the agent's ability to operate on them.
This limited what we can do in integration tests:
1. We have to write test prompts that can operate on generic resources or use follow ups to react to potential questions from the agent asking for Azure scope.
2. We cannot compare the agent outcome with a ground truth answer for correctness. If we can limit what resource the agent needs to operate on, for certain prompts we can pre-compute a ground truth answer to compare the actual output with.
3. Token usage comparison becomes hard. Comparing the token usage on different Azure resources is not an apples to apples comparison.
## Solution
The proposed solution is to establish a standard way for integration test authors to define the Azure fixtures for each integration test, and inject the context information to the LLM context of the integration test to have the agent operate on it. This allows us to test scenarios that require well designed fixtures and allows us to expect determinisitc outcome for the test result.
## Goals
- A standardized way for integration test authors to define Azure fixtures
- Integration test runner (local or CI) will provision the fixtures so the tests have the expected resources to operate on
- Once defined, information of the Azure fixtures are injected into the LLM context for the integration tests to inform the agent what resources to operate on
- The test runner cleans up the provisioned fixtures after test finishes
Contributor guide
Assessment
This issue has not been assessed yet.