Azure / Azure/azure-sdk-tools

Build Evaluation Framework for the SDK Release Agent for E2E flow

Open
#14,826 0 comments 0 reactions 0 assignees View on GitHub
AzSDK Tools Agent Epic P1
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

## Challenges

The SDK release agent is becoming a critical path for self-service releases. As the agent evolves, we need a reliable way to validate behavior.

Today, SH team has been asked to test for the SDK release agent manual which is difficult to test all cases and is unable to scale, there is interest in leveraging an eval approach for covering a scenario set more consistently as the agent changes

## Goal
Build a test and evaluation framework for the SDK release agent that enables
- Repeatable scenario-based testing
- Regression detection for agent changes
- Step evaluation and end-to-end flow evaluation

The Shanghai team can contribute real service-team release scenarios encountered in self-service releases and help convert them into reusable test cases for the scenario catalog.

Contributor guide

Open the contributing guide

Research direction

Start by locating the SDK release agent and reviewing the current manual testing approach described in the issue. Define the scenario catalog, step and end-to-end evaluations, repeatable execution, and regression criteria; done means reusable service-team release scenarios can run consistently and detect agent changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
release, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.