Azure / Azure/azure-sdk-tools

[AzSDK Agent] E2E Validation: Rust — Full workflow with new shared skills

Open
#15,138 0 comments 0 reactions 0 assignees View on GitHub
Scenario 2
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 2h
Merged PRs (30d)
130

Description

## Objective

End-to-end validation of all new shared Copilot skills in the `azure-sdk-for-rust` repo. Test the complete inner loop workflow from TypeSpec to release.

## Steps

### Step 1: Merge the Skills Sync PR

Merge the `.github` directory sync PR to bring the new shared skills into the repo.

- [ ] Merge → Azure/azure-sdk-for-rust#4141 — Sync .github directory (skills)

### Step 2: Update the Instruction Files Sync PR

Update the `eng/common` directory sync PR branch to pull in the latest changes from `main` (which now includes the new skills from Step 1). This ensures the instruction files reference the newly available skills.

- [ ] Update branch on → Azure/azure-sdk-for-rust#4142 — Sync eng/common directory (instructions)

### Step 3: Review for stale instruction file references

After updating the instruction files sync PR branch, review the repo's `.github/copilot-instructions.md` and `AGENTS.md` files to check if they contain references to instruction files that were removed during the instruction-to-skill consolidation. If stale references are found:

- [ ] Remove the stale references, OR
- [ ] Update them to point to the new skill that contains similar information to what the old instruction file had

### Step 4: Run E2E Validation in the Instruction Files Sync PR Branch

Check out the branch from the instruction files sync PR (Azure/azure-sdk-for-rust#4142) and run the full E2E validation there. This validates that the new skills and updated instructions work together correctly.

#### Skills Under Test

| Skill | Stage | Key Action |
|-------|-------|------------|
| `generate-sdk-locally` | Generate → Build → Test | Generate Rust SDK from TypeSpec, build, run tests |
| `apiview-feedback-resolution` | APIView Feedback | Resolve APIView comments on a Rust SDK PR |
| `pipeline-troubleshooting` | Pipeline/CI | Diagnose a Rust CI pipeline failure |
| `prepare-release-plan` | Release Planning | Create and link a release plan for Rust package |
| `sdk-release` | Release | Check readiness and trigger Rust SDK release |

#### E2E Flow to Test

1. Start with a TypeSpec spec for a service
2. Ask the agent to generate the Rust SDK → `generate-sdk-locally`
3. Build and test → `generate-sdk-locally` (build/test stages)
4. Submit for API review → check APIView feedback → `apiview-feedback-resolution`
- **4a. Customizations flow:** After receiving APIView feedback, test resolving comments that require TypeSpec customizations (e.g. `azsdk_typespec_customized_code_update`). Verify the agent applies the customization, regenerates the SDK, rebuilds, and re-submits for review.
5. If pipeline fails → `pipeline-troubleshooting`
6. Create release plan → `prepare-release-plan`
7. Check release readiness → `sdk-release`

#### Key Validation: "What's Next?" Flow

> **Critical test:** At each stage transition, ask `"What's next?"` and verify the agent properly leads to the next logical step in the workflow.

- Does the agent connect the skills into a coherent workflow?
- Does it know which skill/stage comes next?
- Does it provide Rust-specific guidance (cargo build, cargo test, crates.io, etc.)?

### Step 5: Merge the Instruction Files Sync PR

Once E2E validation passes, merge the instruction files sync PR.

- [ ] Merge → Azure/azure-sdk-for-rust#4142 — Sync eng/common directory (instructions)

## Environment

- **Repo:** `Azure/azure-sdk-for-rust`
- **Prerequisites:** Rust toolchain (stable), azure-sdk-mcp server, local clone
- **Language-specific tools:** `cargo build`, `cargo test`, crates.io

## Related Sync PRs (azure-sdk-for-rust)
- Azure/azure-sdk-for-rust#4141 — Sync .github directory (skills)
- Azure/azure-sdk-for-rust#4142 — Sync eng/common directory (instructions)

_Source PR: Azure/azure-sdk-tools#15062_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Azure/azure-sdk-rust#4141 and #4142, then inspect .github/copilot-instructions.md and AGENTS.md for stale references. In the instruction-sync branch, run the listed end-to-end workflow with the azure-sdk-mcp server, including cargo build and cargo test. Done means all five skills work through the Rust workflow, each "What's next?" transition is correct, and the instruction PR is ready to merge.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
developer-experience, testing-qa, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.