microsoft / microsoft/GitHub-Copilot-for-Azure

[Epic] Deterministic integration tests with Azure fixture

Open
#3,139 0 comments 0 reactions 1 assignee Claimed by @JasonYeMSFT View on GitHub
epic
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Problem

Some skills are designed to operate on existing Azure resources, such as azure-resource-lookup, azure-resource-visualizer, azure-diagnostics. As of today, we don't have a well defined solution for providing Azure fixtures to integration tests to evaluate how to affect the agent's ability to operate on them.

This limited what we can do in integration tests:

1. We have to write test prompts that can operate on generic resources or use follow ups to react to potential questions from the agent asking for Azure scope.

2. We cannot compare the agent outcome with a ground truth answer for correctness. If we can limit what resource the agent needs to operate on, for certain prompts we can pre-compute a ground truth answer to compare the actual output with.

3. Token usage comparison becomes hard. Comparing the token usage on different Azure resources is not an apples to apples comparison.

## Solution

The proposed solution is to establish a standard way for integration test authors to define the Azure fixtures for each integration test, and inject the context information to the LLM context of the integration test to have the agent operate on it. This allows us to test scenarios that require well designed fixtures and allows us to expect determinisitc outcome for the test result.

## Goals

- A standardized way for integration test authors to define Azure fixtures
- Integration test runner (local or CI) will provision the fixtures so the tests have the expected resources to operate on
- Once defined, information of the Azure fixtures are injected into the LLM context for the integration tests to inform the agent what resources to operate on
- The test runner cleans up the provisioned fixtures after test finishes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.