microsoft / microsoft/typespec

Speed up end-to-end C# Azure SDK regeneration

Open
#11,361 4 comments 1 reaction 1 assignee Claimed by @live1206 View on GitHub
emitter:client:csharp feature
Dominant language
Java
Stars
5.9k
Forks
394
Avg merge
1d 23h
Merged PRs (30d)
104

Description

## Problem

The provider-based reference map significantly improves C# generator performance, including large management-plane SDKs, but the end-to-end Azure SDK regeneration in `packages/http-client-csharp/eng/pipeline/publish.yml` does not show a comparable wall-clock improvement and can still approach the 90-minute job timeout.

Related work:

- microsoft/typespec#10976
- microsoft/typespec#11288

## Current behavior

The full data-plane + management-plane regeneration currently covers approximately:

- 243 SDK projects
- 202 distinct service directories
- 197 management-plane SDK projects across 188 service directories

`Submit-AzureSdkForNetPr.ps1` processes service directories serially. For every service it starts a new command similar to:

```text
dotnet msbuild eng/service.proj /restore /t:GenerateCode /p:Trace=true /p:ServiceDirectory=
```

This repeats substantial work outside the optimized generator phase:

- MSBuild process startup, evaluation, and restore for each service directory
- restore graph discovery that includes tests, samples, perf, stress, and other non-codegen projects by default
- serial `InstallTspClient` traversal and repeated `npm ci` work
- repeated client plugin builds; child generation receives `SkipTspClientInstall=true` but not `SkipBuildPlugin=true`
- `tsp-client update` and TypeSpec/spec synchronization for every SDK
- trace/debug logging and generated-file disk I/O

The outer service loop itself has no bounded parallelism. The job is capped at 90 minutes, so runs that reach the cap do not provide a useful completed-runtime comparison.

The generator microbenchmark in #10976 improved from 847.4 ms to 544.5 ms. That saves only about 74 seconds when naively multiplied across 243 typical generations, which can be hidden by orchestration variance. The large SDK measurements are much more significant—Network, DataFactory, and AppService together showed nearly 48 minutes of local savings—so a completed post-#11288 full run should still be measurably faster if it exercises the same optimized path.

## Proposed work

1. Add phase and per-service timing with an end-of-job summary that separates setup, restore, tsp-client install/sync, TypeSpec compilation, C# generator execution, plugin build, and file writing.
2. Install tsp-client once per job and reuse it across SDK generation.
3. Build the client plugin once and pass `SkipBuildPlugin=true` to child generation where safe.
4. Restrict restore and MSBuild traversal to actual code-generation projects.
5. Avoid repeating restore/spec synchronization when inputs and dependencies are shared or already prepared.
6. Add bounded service-level parallelism. Resolve shared plugin-output conflicts generically rather than maintaining service-specific serialization exceptions.
7. Re-evaluate trace/debug logging volume in the full regeneration path.

## Acceptance criteria

- Capture a completed baseline and optimized run using the same TypeSpec commit, Azure SDK commit, parameters, agent image, and cache state.
- Report a phase-level timing breakdown and per-service timings in pipeline output.
- The full data-plane + management-plane regeneration completes comfortably below the 90-minute timeout with a clear wall-clock improvement over baseline.
- Generated output is identical to the equivalent sequential regeneration.
- Parallel execution does not introduce shared-output races or flaky generation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.