dotnet / dotnet/sdk

Multi-arch container publishing conflicts with the `--output` option in `dotnet publish`

Open
#46,109 4 comments 4 reactions 1 assignee Claimed by @baronfel View on GitHub
Area-Containers Area-NetSDK
Dominant language
C#
Stars
3.2k
Forks
1.3k
PR merge metrics
PR metrics pending

Description

### Describe the bug

Invoking a multi-arch container publish of a project while also overriding the `PublishDir` results in the per-RID inner builds all having the same `PublishDir`. This results in race conditions during the build, and if there are any native, RID-specific assets in the build can result in mixing assets for different platforms.

### To Reproduce

Using .NET SDK 9.0.102 or .NET SDK 8.0.405 or later,

* `dotnet new webapi`
* Add `linux-x64;linux-arm64`
* run `dotnet publish -t:PublishContainer --output `
* watch the MSB3026 warnings fly:

```
>dotnet publish -t:PublishContainer -v d -bl -o bugsville
Restore complete (0.5s)
Determining projects to restore...
Restored E:\Code\Scratch\multi-arch-build-bug\multi-arch-build-bug.csproj (in 217 ms).
C:\Program Files\dotnet\sdk\9.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(355,5): warning MSB3026: Could not copy "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json" to "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json". Beginning retry 1 in 1000ms. The process cannot access the file 'E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json' because it is being used by another process.
C:\Program Files\dotnet\sdk\9.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(355,5): warning MSB3026: Could not copy "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json" to "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json". Beginning retry 1 in 1000ms. The process cannot access the file 'E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json' because it is being used by another process.
multi-arch-build-bug succeeded (3.5s) → bugsville\
Building image 'multi-arch-build-bug' with tags 'latest-linux-x64' on top of base image 'mcr.microsoft.com/dotnet/aspnet:9.0'.
Pushed image 'multi-arch-build-bug:latest-linux-x64' to local registry via 'docker'.
multi-arch-build-bug succeeded with 1 warning(s) (5.4s) → bugsville\
C:\Program Files\dotnet\sdk\9.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(355,5): warning MSB3026: Could not copy "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json" to "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json". Beginning retry 1 in 1000ms. The process cannot access the file 'E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json' because it is being used by another process.
Building image 'multi-arch-build-bug' with tags 'latest-linux-arm64' on top of base image 'mcr.microsoft.com/dotnet/aspnet:9.0'.
Pushed image 'multi-arch-build-bug:latest-linux-arm64' to local registry via 'docker'.
multi-arch-build-bug succeeded with 1 warning(s) (4.2s) → bugsville\
C:\Program Files\dotnet\sdk\9.0.102\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Publish.targets(355,5): warning MSB3026: Could not copy "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json" to "E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json". Beginning retry 1 in 1000ms. The process cannot access the file 'E:\Code\Scratch\multi-arch-build-bug\bugsville\bugsville\bugsville\multi-arch-build-bug.runtimeconfig.json' because it is being used by another process.

Build succeeded with 2 warning(s) in 11.0s
```

### Details

What's happening here is that the `--output` CLI arg is passing the `PublishDir` global property with a value of `bugsville` to MSBuild. This is used for the 'outer' publish successfully, but because a multi-arch container publish is triggered, we do two RID-specific inner publishes _in parallel_. These are done with the same values of PublishDir, which causes the conflicts.

### Potential solutions

* provide a knob to stop the parallel inner builds
* use `AdditionalProperties` on the inner builds to set a new value of PublishDir
* have `--output` mean a different property for a multi-RID build and define what PublishDir should be in terms of that property
* I'm sure there are other options
* Set `RemoveProperties` on the `MSBuild` Task invocations for the inner builds

### Commentary

This is coming up because, unlike for multi-TFM builds, we're not blocking publishing for multiple RIDs. The Publish target for multi-targeted builds even says:
```
Currently it is unsupported to publish for multiple target frameworks
because users can specify the $(PublishDir), and publish would put
multiple published applications in a single directory.
```

Yet here we are.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.