microsoft / microsoft/typespec

[http-client-csharp] Parallel Azure SDK regeneration races on shared MSBuild intermediate files

Open
#11,822 0 comments 2 reactions 2 assignees Claimed by @JoshLove-msft View on GitHub
bug emitter:client:csharp
Dominant language
Java
Stars
5.9k
Forks
394
Avg merge
1d 23h
Merged PRs (30d)
104

Description

## Description

Parallel Azure SDK regeneration can fail nondeterministically when multiple library builds restore shared projects into the same MSBuild intermediate directory. This has occurred more than once and makes the `typespec - http-client-csharp - publish` regeneration pipeline flaky.

The most recent occurrence was internal build [6776805](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6776805), triggered for microsoft/typespec#11818.

## Observed failure

The pipeline dispatched 49 regeneration jobs with concurrency 12:

```text
Dispatching 49 regeneration jobs (12 at a time)...
```

`Azure.Communication.JobRouter` then failed during restore, before generator behavior was exercised:

```text
NuGet.targets(198,5): error : The file
'.../artifacts/obj/Azure.SdkAnalyzers/Azure.SdkAnalyzers.csproj.nuget.g.props'
already exists.
```

The run completed with:

- 48 libraries successfully regenerated
- 1 actual failure: `Azure.Communication.JobRouter`
- 4 libraries skipped because an earlier regeneration batch failed

The same PR's normal Linux and Windows C# regeneration checks passed, further indicating that this was an orchestration race rather than a generator regression.

## Likely cause

`Invoke-SdkLibraryRegeneration` starts independent library builds concurrently. Those builds can restore common projects such as `Azure.SdkAnalyzers` while sharing the repository-level `artifacts/obj` directory. Concurrent restore processes then race while creating the same generated NuGet files.

The error is consistent with two restore operations attempting to create:

```text
artifacts/obj/Azure.SdkAnalyzers/Azure.SdkAnalyzers.csproj.nuget.g.props
```

## Expected behavior

Parallel regeneration should isolate or coordinate shared restore/build outputs so independent libraries cannot write the same intermediate files concurrently. A transient restore race should also not abort all remaining regeneration batches without a retry.

Potential approaches include:

- perform a single restore of shared projects before dispatching parallel generation;
- use isolated MSBuild intermediate paths per regeneration worker;
- avoid restoring shared analyzer/tool projects independently in each worker;
- serialize the restore phase while keeping generation parallel; or
- retry failures that match known transient file-creation conflicts after concurrent work settles.

## Acceptance criteria

- Repeated parallel full-regeneration runs do not fail with shared `artifacts/obj` file conflicts.
- Parallel workers do not concurrently mutate the same MSBuild/NuGet intermediate files.
- Generated output remains identical to sequential regeneration.
- A focused test or pipeline stress validation exercises concurrent restoration of libraries sharing `Azure.SdkAnalyzers`.

This is related to the race-free parallelism requirement in #11361, but is tracked separately as a concrete recurring pipeline bug.

- by copilot

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.