[Enhancement]: BuildKit support for `ImageFromDockerfileBuilder` via a containerized `docker buildx` builder

Problem

Image builds go through POST /build on the Docker Engine API (DockerImageOperations.BuildAsync), which uses the legacy builder. Any Dockerfile that depends on BuildKit cannot be built with Testcontainers, even when it builds fine with docker build on the same host. This is documented as a limitation in docs/api/create_docker_image.md and has produced several issues:

• #1247: heredoc (RUN cat <<EOF) silently produces an empty file, and the container then fails to start with exec format error
• #1406: no way to pass build secrets (--secret, RUN --mount=type=secret)
• Related: $BUILDPLATFORM (#993), and anything else behind # syntax=docker/dockerfile:1.x

The position so far (#1193, discussioncomment-10315903) is that BuildKit is not part of the Docker Engine API, and that support would be added as soon as it is possible or someone comes up with a workaround. This is a proposal for such a workaround, using the same approach the new Compose support already uses.

Solution

Opt-in BuildKit builds run by the Docker CLI inside a container, following the pattern from Compose support (#1750):

1. A keep-alive docker:28-cli container. That image ships the buildx plugin. Started the way ComposeBuilder starts its container (src/Testcontainers/Builders/ComposeBuilder.cs:386-387).
2. The Docker socket bind-mounted read-only through the existing UnixSocketMount (ComposeBuilder.cs:374), so the build runs against the same daemon as the rest of the test session.
3. The build context copied into the container with resource mapping, the same way compose files are (ComposeBuilder.cs:377). The tarball that ITarArchive.Tar already produces can be reused here.
4. docker buildx build --load invoked with ExecAsync, the same shape as ComposeContainer.ComposeUpCommand (src/Testcontainers/Containers/ComposeContainer.cs:51,409).

With --load the result is written to the daemon's image store, so everything after the build behaves as it does today: image name resolution, WithImage, and resource reaper labels passed as --label. The existing IImageFromDockerfileConfiguration maps onto CLI flags directly: Dockerfile to --file, Image to --tag, BuildArguments to --build-arg, Labels to --label, Target to --target, plus new options for --secret, --ssh and --platform.

The Engine API path stays the default and is not touched. BuildKit is enabled explicitly (something like WithBuildKit(), exact API up for discussion). A first PR would cover heredoc (#1247) and build secrets (#1406).

Verified locally

Docker 29.1.3, docker:28-cli with buildx v0.29.1, daemon BuildKit v0.26.2, socket mounted read-only, context copied in with docker cp, then docker buildx build --load inside the container. Results:

• The image is written to the host daemon's image store and is visible to docker images outside the build container.
• Heredoc produces the expected file contents, and the entrypoint runs.
• --secret id=mysecret,src=... is readable at /run/secrets/mysecret during RUN, and /run/secrets is not present in the final image.
• Both a --label and a LABEL fed by --build-arg (the org.testcontainers.resource-reaper-session pattern from the docs) are set on the resulting image.

Questions before I write any code

1. API shape: a flag on ImageFromDockerfileBuilder, or a separate builder type? Should the CLI image be configurable and pinned the way ComposeBuilder(string image) is?
2. Non-socket endpoints: TLS/TCP hosts and Windows named pipes cannot be bind-mounted. Fall back to the legacy builder with a warning, or throw?
3. Secrets: which sources to support (file, environment variable, in-memory), and making sure secret values reach only the builder container, never the context tarball or the image.
4. Builder lifecycle: one build container per build or a shared one, and whether buildx cache should be reused across builds at all in a test run.
5. Longer term: would you rather aim for native BuildKit session support in testcontainers/Docker.DotNet (POST /build?version=2&session=<id> plus the gRPC session for filesync, auth and secrets), with this as an interim step? Since the client is a Testcontainers-owned fork that route is not blocked upstream, it is just a lot more work.

Benefit

• Closes #1247 and #1406, and allows Dockerfiles that use # syntax= frontends, RUN --mount, and platform build args.
• Removes a recurring source of confusing failures where a Dockerfile builds with the CLI but not with Testcontainers. In #1247 the build even reports success and produces a broken image.
• Reuses code that already exists and is already tested, instead of adding a dependency or a new execution model.
• Keeps Testcontainers from being a CLI wrapper. Nothing runs on the test host, only inside a container the library already manages.

Alternatives

1. Call the host docker CLI. Simplest option, but it breaks the API-only design and depends on whatever CLI and buildx version the host happens to have. Not proposed.
2. Native BuildKit session support in Docker.DotNet. Best fit with the current architecture, but it needs BuildKit protobufs and a gRPC session over a hijacked connection in .NET, and it holds up both open issues until that is done.
3. Leave it as is and document workarounds. Build the image with the CLI outside the test and pass the name in, or replace heredocs with WithResourceMapping and WithStartupCallback. Both work, but they move build orchestration out of the test and neither helps with secrets.
4. Use a moby/buildkit container instead of docker:28-cli, so a real buildkitd rather than the daemon's builder. More control, but a heavier moving part and further from the Compose approach.

Would you like to help contributing this enhancement?

Yes

Start with ImageFromDockerfileBuilder and DockerImageOperations.BuildAsync, then compare ComposeBuilder.cs, ComposeContainer.cs, UnixSocketMount, and ITarArchive.Tar as referenced in the proposal. Review the existing image-build tests and Compose support before resolving the API, endpoint, secret, and lifecycle questions. Done means opt-in BuildKit builds support the agreed options while the legacy path remains default and heredoc and secret scenarios pass.