long-running mutistage build gets canceled around 1min if stages are from `mcr.microsoft.com`

## TL;DR:
1. skaffold build (direct `docker build` succeed)
2. docker backend (regardless of buildkit or not)
3. multistage Dockerfile
4. all stages are from `mcr.microsoft.com` (other registries have no problem)
5. the first stage takes more than 60s
6. then it gets canceled around 1 min or at the start of the second stage due to the internal EPIPE error that causes SIGPIPE.

## detailed explaination, reproduction, eliminations:

We've been having this problem since a few days ago (first occurrence: 2026-06-05, widespread: 2026-06-08)

### minimal reproduce

If we have the following `Dockerfile`:

```dockerfile
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
RUN mkdir /src/ && touch /src/asdf && sleep 90

FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS aspnet
RUN mkdir /app/
COPY --from=sdk /src/asdf /app/asdf
```

and the following `skaffold.yaml`:
```yaml
apiVersion: skaffold/v4beta11
kind: Config
metadata:
name: test
build:
artifacts:
- image: test
local:
push: false
```

and run the following command:
```shell
docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
skaffold build
```

The build gets canceled in the middle of the build (after around 1 min)
```
Generating tags...
- test -> test:latest
Some taggers failed. Rerun with -vdebug for errors.
Checking cache...
- test: Not found. Building
Starting build...
Building [test]...
Sending build context to Docker daemon 2.048kB
Step 1/5 : FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
---> c07906393326
Step 2/5 : RUN mkdir /src/ && touch /src/asdf && sleep 90
---> Running in 25dce5cfedc2
Build [test] was canceled
```

This is the last few lines of logs with `-v trace`

```
---8<--- snip

DEBU[0000] Could not import artifact from Docker, building instead (import of missing images disabled) subtask=-1 task=Build
- test: Not found. Building
INFO[0000] Cache check completed in 306.481713ms subtask=-1 task=Build
Starting build...
Building [test]...
DEBU[0000] Running docker build: context: ., dockerfile: Dockerfile subtask=test task=Build
DEBU[0000] Skipping credential configuration because docker-credential-gcloud is not on PATH. subtask=-1 task=DevLoop
INFO[0000] Creating tar file from 1 file(s) subtask=test task=Build
INFO[0000] Creating tar file completed in 112.382µs subtask=test task=Build
Sending build context to Docker daemon 2.048kB
Step 1/5 : FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
---> c07906393326
Step 2/5 : RUN mkdir /src/ && touch /src/asdf && sleep 90
---> Running in 9d744bb3a429
Build [test] was canceled

DEBU[0066] exporting metrics subtask=-1 task=DevLoop
DEBU[0066] failed to create firelog exporter due to error: open assets/firelog_generated/key.txt: file does not exist subtask=-1 task=DevLoop
DEBU[0066] did not create any log exporter subtask=-1 task=DevLoop

---8<--- snip
```

### it happens regardless of BuildKit.

this is the last lines of logs with following `skaffold.yaml` to enable buildkit
```yaml
apiVersion: skaffold/v4beta11
kind: Config
metadata:
name: test
build:
artifacts:
- image: test
local:
push: false
useBuildkit: true
```

```
---8<--- snip

DEBU[0000] Found dependencies for dockerfile: [] subtask=-1 task=DevLoop
DEBU[0000] Could not import artifact from Docker, building instead (import of missing images disabled) subtask=-1 task=Build
- test: Not found. Building
INFO[0000] Cache check completed in 306.195482ms subtask=-1 task=Build
Starting build...
Building [test]...
DEBU[0000] Running command: [docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest --load] subtask=test task=Build
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 237B done
#1 DONE 0.0s

#2 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:10.0
#2 DONE 0.0s

#3 [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:10.0
#3 DONE 0.1s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [aspnet 1/3] FROM mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711
#5 resolve mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711 0.0s done
#5 DONE 0.2s

#6 [sdk 1/2] FROM mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6
#6 resolve mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6 0.0s done
#6 DONE 0.3s

#7 [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90
#7 ...

#8 [aspnet 2/3] RUN mkdir /app/
#8 DONE 0.2s

#7 [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90
#7 CANCELED
ERROR: failed to build: failed to solve: Canceled: context canceled
Build [test] was canceled

DEBU[0066] exporting metrics subtask=-1 task=DevLoop
DEBU[0066] failed to create firelog exporter due to error: open assets/firelog_generated/key.txt: file does not exist subtask=-1 task=DevLoop
DEBU[0066] did not create any log exporter subtask=-1 task=DevLoop

---8<--- snip
```

### It happens only with skaffold + docker, not with just docker, nor skaffold + podman

If I run docker command in the log directly, it finishes the build successfully
```
docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest --load
```

```
[+] Building 91.1s (10/10) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 237B 0.0s
=> [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:10.0 0.1s
=> [internal] load metadata for mcr.microsoft.com/dotnet/sdk:10.0 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [sdk 1/2] FROM mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6 0.2s
=> => resolve mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6 0.0s
=> [aspnet 1/3] FROM mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711 0.1s
=> => resolve mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711 0.0s
=> [aspnet 2/3] RUN mkdir /app/ 0.2s
=> [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90 90.3s
=> [aspnet 3/3] COPY --from=sdk /src/asdf /app/asdf 0.0s
=> exporting to image 0.3s
=> => exporting layers 0.1s
=> => exporting manifest sha256:c7128685369cee3f5533b74b0faf3e10a0e26393c36ad1bb5efa0235c23d64d7 0.0s
=> => exporting config sha256:b7662798278842ddbfcea6bd09a635ef9fda770e3f9511e9bbe9d7a70cae1915 0.0s
=> => exporting attestation manifest sha256:859bebcb0743d1aa4693bacc22d2882627db4bdb5bb4281389e71e9f2b6867d7 0.0s
=> => exporting manifest list sha256:3c93970a8c7e1b725725aaf7b5c272446154c681bf0c258d0fab3b3920ed3acf 0.0s
=> => naming to docker.io/library/test:latest 0.0s
=> => unpacking to docker.io/library/test:latest 0.0s
```

it also build successfully without the buildkit:
```
docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
DOCKER_BUILDKIT=0 docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest
```

```
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

Sending build context to Docker daemon 3.072kB
Step 1/5 : FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
---> c07906393326
Step 2/5 : RUN mkdir /src/ && touch /src/asdf && sleep 90
---> Running in 47919a2bd636
---> Removed intermediate container 47919a2bd636
---> ed6003243083
Step 3/5 : FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS aspnet
10.0: Pulling from dotnet/aspnet
Digest: sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711
Status: Downloaded newer image for mcr.microsoft.com/dotnet/aspnet:10.0
---> 8c0b6857eab7
Step 4/5 : RUN mkdir /app/
---> Running in b3c53e899f05
---> Removed intermediate container b3c53e899f05
---> 0a2bc21fcfeb
Step 5/5 : COPY --from=sdk /src/asdf /app/asdf
---> 93dd0fe9c9d1
Successfully built 93dd0fe9c9d1
Successfully tagged test:latest
```

~Also it builds successfully with podman backend.~
podman also fails.

### It happens only with `mcr.microsoft.com`

If we change the Dockerfile to images from another registry, Skaffold has no problem with it.
```
FROM redis AS sdk
RUN mkdir /src/ && touch /src/asdf && sleep 90
FROM busybox AS aspnet
RUN mkdir /app/
COPY --from=sdk /src/asdf /app/asdf
```

```shell
docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
skaffold build
```

```
---8<--- snip

DEBU[0002] Running command: [docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest --load] subtask=test task=Build
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 180B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/redis:latest
#2 DONE 1.1s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 1.1s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [aspnet 1/3] FROM docker.io/library/busybox:latest@sha256:fd8d9aa63ba2f0982b5304e1ee8d3b90a210bc1ffb5314d980eb6962f1a9715d
#5 resolve docker.io/library/busybox:latest@sha256:fd8d9aa63ba2f0982b5304e1ee8d3b90a210bc1ffb5314d980eb6962f1a9715d 0.0s done
#5 DONE 0.1s

#6 [sdk 1/2] FROM docker.io/library/redis:latest@sha256:aa049e689e141a4358ad1d4562dc49c88a89fbab711fd8fcc33f684c80b26301
#6 resolve docker.io/library/redis:latest@sha256:aa049e689e141a4358ad1d4562dc49c88a89fbab711fd8fcc33f684c80b26301 0.0s done
#6 DONE 0.2s

#7 [aspnet 2/3] RUN mkdir /app/
#7 DONE 0.2s

#8 [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90
#8 DONE 90.2s

#9 [aspnet 3/3] COPY --from=sdk /src/asdf /app/asdf
#9 DONE 0.0s

#10 exporting to image
#10 exporting layers 0.1s done
#10 exporting manifest sha256:32e100d0f1739ade7d4a5cae261fa1d95789af17ef5d7307c13fccfd7e2bd6ac 0.0s done
#10 exporting config sha256:d40e11d6da493d7b9f22626d04ce044f1db3d28d3368e3a7d83a172e8dc51eed 0.0s done
#10 exporting attestation manifest sha256:1eb9a12fd3d3224f2c1d3186e361adf6dd64f2d99804d83539128be317afe89d 0.0s done
#10 exporting manifest list sha256:eea4c7323409f42196b751585776c8984f1eab41b67003369f7156c9a494cd77 0.0s done
#10 naming to docker.io/library/test:latest
#10 naming to docker.io/library/test:latest done
#10 unpacking to docker.io/library/test:latest 0.0s done
#10 DONE 0.3s
Build [test] succeeded
INFO[0095] Build completed in 1 minute 32.166 seconds subtask=-1 task=Build

DEBU[0095] exporting metrics subtask=-1 task=DevLoop
DEBU[0095] failed to create firelog exporter due to error: open assets/firelog_generated/key.txt: file does not exist subtask=-1 task=DevLoop
DEBU[0095] did not create any log exporter subtask=-1 task=DevLoop

---8<--- snip
```

## findings

The docker command is killed by skaffold. skaffold kills the docker command when it got `SIGPIPE` signal.
The cancel is from this context, and it has the reason `broken pipe signal received`
https://github.com/GoogleContainerTools/skaffold/blob/f9beeb7b68778673806c5e9a3b5748af916116cd/cmd/skaffold/app/skaffold.go#L36

strace shows that it got SIGPIPE with microsoft's IP address at the start of second stage.

```shell
docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
strace -f -yy --trace=read,write,rt_sigprocmask,rt_sigreturn skaffold build
```

```
---8<--- snip

[pid 223504] read(8150.171.69.10:443]>, 0xc00001cf00, 9472) = -1 ECONNRESET (Connection reset by peer)
[pid 223504] write(8150.171.69.10:443]>, "\27\3\3\0\23+,\334h|\224!]\277\16$4*\243:\2015v\351", 24) = -1 EPIPE (Broken pipe)
[pid 223504] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=223450, si_uid=1000} ---
[pid 223504] rt_sigreturn({mask=[]}) = -1 EPIPE (Broken pipe)

---8<--- snip

[pid 223547] write(2, "#7 CANCELED\n", 12
[pid 223544] read(103132335]>
[pid 223547] <... write resumed>) = 12
[pid 223544] <... read resumed>, 0xc000802000, 32768) = -1 EAGAIN (Resource temporarily unavailable)
[pid 223456] read(10, "#7 CANCELED\n", 32768) = 12
[pid 223456] write(1>, "#7 CANCELED\n", 12#7 CANCELED
) = 12

---8<--- snip
```

I couldn't find SIGPIPE when I invoked the Docker CLI directly

```shell
docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
DOCKER_BUILDKIT=0 strace -f -yy --trace=read,write,rt_sigprocmask,rt_sigreturn docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest
```

## environment:

multiple version of skaffold:

on our CI environment ->
OS: ubuntu 24.04
skaffold: v2.16.1, v2.21.0, v2.22.0, v2.22.1
docker:
```
Client:
Version: 29.3.0
API version: 1.54
Go version: go1.25.7
Git commit: 5927d80
Built: Thu Mar 5 14:24:57 2026
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 29.3.0-namespace
API version: 1.54 (minimum version 1.40)
Go version: go1.25.7
Git commit: HEAD
Built: Wed Apr 22 16:26:32 2026
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 2.2.1+unknown
GitCommit:
runc:
Version: 1.3.4
GitCommit: v1.3.4-0-gd6d73eb
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```

one of my team member's workstation ->
OS: nix OS
skaffold: ?
docker:
```
Client:
Version: 28.5.2
API version: 1.51
Go version: go1.25.9
Git commit: v28.5.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 28.5.2
API version: 1.51 (minimum version 1.24)
Go version: go1.25.9
Git commit: v28.5.2
Built: Tue Jan 1 00:00:00 1980
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v2.2.1
GitCommit: refs/tags/v2.2.1
runc:
Version: 1.3.3
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
```

one of my other team member's workstation ->
OS: WSL2
skaffold: ?
docker:
```
Client:
Version: 29.5.2
API version: 1.54
Go version: go1.26.3
Git commit: 79eb04c
Built: Wed May 20 14:37:09 2026
OS/Arch: linux/amd64
Context: default

Server: Docker Desktop 4.76.0 (228118)
Engine:
Version: 29.5.2
API version: 1.54 (minimum version 1.40)
Go version: go1.26.3
Git commit: 568f755
Built: Wed May 20 14:38:09 2026
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v2.2.4
GitCommit: 193637f7ee8ae5f5aa5248f49e7baa3e6164966e
runc:
Version: 1.3.5
GitCommit: v1.3.5-0-g488fc13e
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```

my workstation ->
OS: ubuntu 26.04
skaffold: v2.14.0-14-ga65fe4bb5
docker:
```
Client:
Version: 29.1.3
API version: 1.52
Go version: go1.24.13
Git commit: 29.1.3-0ubuntu4.1
Built: Wed Apr 29 16:40:20 2026
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 29.1.3
API version: 1.52 (minimum version 1.44)
Go version: go1.24.13
Git commit: 29.1.3-0ubuntu4.1
Built: Wed Apr 29 16:40:20 2026
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 2.2.2
GitCommit:
runc:
Version: 1.4.0-0ubuntu1
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
```

## workaround:

pulling all images before the skaffold build prevents this problem
```
docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
docker image pull mcr.microsoft.com/dotnet/sdk:10.0
docker image pull mcr.microsoft.com/dotnet/aspnet:10.0
skaffold build
```