microsoft / microsoft/typespec
[http-client-csharp] Parallel Azure SDK regeneration races on shared MSBuild intermediate files
- Dominant language
- Java
- Stars
- 5.9k
- Forks
- 394
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 104
Description
## Description
Parallel Azure SDK regeneration can fail nondeterministically when multiple library builds restore shared projects into the same MSBuild intermediate directory. This has occurred more than once and makes the `typespec - http-client-csharp - publish` regeneration pipeline flaky.
The most recent occurrence was internal build [6776805](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6776805), triggered for microsoft/typespec#11818.
## Observed failure
The pipeline dispatched 49 regeneration jobs with concurrency 12:
```text
Dispatching 49 regeneration jobs (12 at a time)...
```
`Azure.Communication.JobRouter` then failed during restore, before generator behavior was exercised:
```text
NuGet.targets(198,5): error : The file
'.../artifacts/obj/Azure.SdkAnalyzers/Azure.SdkAnalyzers.csproj.nuget.g.props'
already exists.
```
The run completed with:
- 48 libraries successfully regenerated
- 1 actual failure: `Azure.Communication.JobRouter`
- 4 libraries skipped because an earlier regeneration batch failed
The same PR's normal Linux and Windows C# regeneration checks passed, further indicating that this was an orchestration race rather than a generator regression.
## Likely cause
`Invoke-SdkLibraryRegeneration` starts independent library builds concurrently. Those builds can restore common projects such as `Azure.SdkAnalyzers` while sharing the repository-level `artifacts/obj` directory. Concurrent restore processes then race while creating the same generated NuGet files.
The error is consistent with two restore operations attempting to create:
```text
artifacts/obj/Azure.SdkAnalyzers/Azure.SdkAnalyzers.csproj.nuget.g.props
```
## Expected behavior
Parallel regeneration should isolate or coordinate shared restore/build outputs so independent libraries cannot write the same intermediate files concurrently. A transient restore race should also not abort all remaining regeneration batches without a retry.
Potential approaches include:
- perform a single restore of shared projects before dispatching parallel generation;
- use isolated MSBuild intermediate paths per regeneration worker;
- avoid restoring shared analyzer/tool projects independently in each worker;
- serialize the restore phase while keeping generation parallel; or
- retry failures that match known transient file-creation conflicts after concurrent work settles.
## Acceptance criteria
- Repeated parallel full-regeneration runs do not fail with shared `artifacts/obj` file conflicts.
- Parallel workers do not concurrently mutate the same MSBuild/NuGet intermediate files.
- Generated output remains identical to sequential regeneration.
- A focused test or pipeline stress validation exercises concurrent restoration of libraries sharing `Azure.SdkAnalyzers`.
This is related to the race-free parallelism requirement in #11361, but is tracked separately as a concrete recurring pipeline bug.
- by copilot
Contributor guide
Assessment
This issue has not been assessed yet.