BuildKit should not terminate parallel stages if one stage fails

I ran into this issue a couple of times (likely due to GitHub flakiness); when building the https://github.com/moby/moby Dockerfile (which runs many stages in parallel), one of the stages ran into a connection issue. If a failure happens in one stage, BuildKit appears to (forcibly?) terminate other stages, even if those stages are still running successfully.

A result of this was that (forcibly) terminating other stages caused an unclean shutdown of the process (git in this case). The stages that were terminated left behind state (index.lock or shallow.lock), which gets persisted between builds because those stages use a cache-mount.

Reproduction example

This was when running an integration test on Docker Desktop (Docker Engine 24.0.2-38-g8e70a1b23e with containerd integration enabled);

make TEST_FILTER=TestStopContainerWithTimeoutCancel DOCKER_GRAPHDRIVER=vfs TESTDEBUG=1 test-integration

During the build, the swagger-src stage failed to fetch from GitHub (this may be due to too many stages running in parallel, but not sure);

------
 > [swagger-src 3/3] RUN git fetch -q --depth 1 origin "c56166c036004ba7a3a321e5951ba472b9ae298c" && git checkout -q FETCH_HEAD:
#0 2.354 error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
#0 2.354 error: 4965 bytes of body are still expected
#0 2.358 fetch-pack: unexpected disconnect while reading sideband packet
#0 2.360 fatal: early EOF
#0 2.360 fatal: index-pack failed
------
Dockerfile:100
--------------------
  98 |     # install. Go-swagger is used in CI for validating swagger.yaml in hack/validate/swagger-gen
  99 |     ARG GO_SWAGGER_COMMIT=c56166c036004ba7a3a321e5951ba472b9ae298c
 100 | >>> RUN git fetch -q --depth 1 origin "${GO_SWAGGER_COMMIT}" && git checkout -q FETCH_HEAD
 101 |
 102 |     FROM base AS swagger
--------------------
ERROR: failed to solve: process "/bin/sh -c git fetch -q --depth 1 origin \"${GO_SWAGGER_COMMIT}\" && git checkout -q FETCH_HEAD" did not complete successfully: exit code: 128

Trying again, after the failed build, I now ran into the build failing again, due to .lock files being left behind after an unclean shutdown;

 > [dockercli-integration 3/3] RUN --mount=type=cache,id=dockercli-integration-git-linux/arm64/v8,target=./.git     --mount=type=cache,target=/root/.cache/go-build,id=dockercli-integration-build-linux/arm64/v8     /download-or-build-cli.sh v17.06.2-ce https://github.com/docker/cli.git /build:
#0 2.410 fatal: Unable to create '/go/src/github.com/docker/cli/.git/index.lock': File exists.
#0 2.410
#0 2.410 Another git process seems to be running in this repository, e.g.
#0 2.410 an editor opened by 'git commit'. Please make sure all processes
#0 2.410 are terminated then try again. If it still fails, a git process
#0 2.410 may have crashed in this repository earlier:
#0 2.410 remove the file manually to continue.

Or:

------
 > [dockercli-integration 3/3] RUN --mount=type=cache,id=dockercli-integration-git-linux/arm64/v8,target=./.git     --mount=type=cache,target=/root/.cache/go-build,id=dockercli-integration-build-linux/arm64/v8     /download-or-build-cli.sh v17.06.2-ce https://github.com/docker/cli.git /build:
#0 1.575 fatal: Unable to create '/go/src/github.com/docker/cli/.git/shallow.lock': File exists.
#0 1.575
#0 1.575 Another git process seems to be running in this repository, e.g.
#0 1.575 an editor opened by 'git commit'. Please make sure all processes
#0 1.575 are terminated then try again. If it still fails, a git process
#0 1.575 may have crashed in this repository earlier:
#0 1.575 remove the file manually to continue.
------
Dockerfile:268
--------------------
 267 |     ARG TARGETPLATFORM
 268 | >>> RUN --mount=type=cache,id=dockercli-integration-git-$TARGETPLATFORM,target=./.git \
 269 | >>>     --mount=type=cache,target=/root/.cache/go-build,id=dockercli-integration-build-$TARGETPLATFORM \
 270 | >>>     /download-or-build-cli.sh ${DOCKERCLI_INTEGRATION_VERSION} ${DOCKERCLI_INTEGRATION_REPOSITORY} /build
 271 |
--------------------
ERROR: failed to solve: process "/bin/sh -c /download-or-build-cli.sh ${DOCKERCLI_INTEGRATION_VERSION} ${DOCKERCLI_INTEGRATION_REPOSITORY} /build" did not complete successfully: exit code: 128

Workarounds

I tried to work around this issue by cleaning the build-cache (docker builder prune), however, doing so (this may be due to too many stages running in parallel, but not sure) another build-stage may now be hitting the connection issue on GitHub, causing the problem to start all over again (besides purging caches). Another workaround (which I'll likely implement for the time being) is to always purge .lock files at the start of the stage; this may be a good solution to prevent the side-effects of the problem, but it would be nice to avoid this situation from the start.

Proposal: don't terminate parallel stages if one stage fails

While the build should ultimately fail if one of the stages fails, I think it would be nice if we would not cancel other builds that are still running; those stages should still be valid (and can be continued to completion). Doing so would help not having to rebuild those stages "from scratch" after a failure, and can prevent bad state in any of those stages.

Start by reproducing the failure with make TEST_FILTER=TestStopContainerWithTimeoutCancel DOCKER_GRAPHDRIVER=vfs TESTDEBUG=1 test-integration, using the reported parallel Dockerfile stages and swagger-src fetch failure. Trace how the failing stage affects other running stages and inspect the cache-mounted Git state; done means the failure behavior and resulting lock-file state are covered by a reliable test.