[Bug]: Building a container when running inside another one with mounted socket is not possible

Module

Core

Testcontainers version

1.19.0

Using the latest Testcontainers version?

Yes

Host OS

RHEL8

Host Arch

x86_64

Docker version

# podman version
Client:       Podman Engine
Version:      4.6.1
API Version:  4.6.1
Go Version:   go1.20.6
Built:        Mon Oct  9 18:15:55 2023
OS/Arch:      linux/amd64

Server:       Podman Engine
Version:      4.6.1
API Version:  4.6.1
Go Version:   go1.20.6
Built:        Mon Oct  9 18:15:55 2023
OS/Arch:      linux/amd64

What happened?

I have an Azure CI/CD pipeline building a Java/Kotlin project using Maven. The Azure Agent itself is deployed as a container and spawns new containers for the builds on the fly.

container:
  image: registry/buildImageABC:1.2
  env:
    TESTCONTAINERS_HOST_OVERRIDE: "host.containers.internal"

Inside this container a maven build consisting of multiple lifecycle phases is running. The one I am having troubles with is the verify phase, where integration tests are supposed to be running. Before verify starts the .jar file is available and packaged.

The part of the integration test that fails looks like this:

GenericContainer<ContainerImageClass>(
                ImageFromDockerfile()
                    .withFileFromPath("Dockerfile", Paths.get("Dockerfile"))
                    .withFileFromPath("target/project.jar", Paths.get("target/project.jar"))
            ).withLogConsumer(Slf4jLogConsumer(logger)).waitingFor(Wait.forHealthcheck()).start()

The Dockerfile looks like this:

FROM baseImage:123
FROM otherBaseImage:456
COPY --from=0 /work /app/work

WORKDIR /app/work
COPY target/*.jar ./

HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=20s \
  CMD curl -f http://127.0.0.1:8080/ || exit 1

I have two intuitions what the problem could be, but no idea how to fix them nicely:

1. the .jar Archive is still open when testcontainers tries to transfer it to the mounted socket. However, the .jar should be built after package, before verify.
2. the podman daemon running on the outside of all containers (on the host machine) is not able to resolve the path /__w/3/s/target/project-0.0.1-DEV.tar which is only valid inside the build container. However, to my understanding testcontainers sends the files to the socket in binary format to create a build context, hence the paths do not need to be present on the host.

Any ideas? Or are some of my intuitions false? Or is this a known bug?

Relevant log output

When running mvn clean verify I'm faced with the following error log:

[docker-java-stream-791735816] ERROR c.g.d.a.async.ResultCallbackTemplate - Error during callback
java.lang.RuntimeException: java.io.IOException: Broken pipe
        at com.github.dockerjava.zerodep.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:195)
        at com.github.dockerjava.zerodep.ZerodepDockerHttpClient.execute(ZerodepDockerHttpClient.java:8)
        at org.testcontainers.dockerclient.HeadersAddingDockerHttpClient.execute(HeadersAddingDockerHttpClient.java:23)
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
...

and

[main] ERROR o.t.utility.MountableFile - Error when copying TAR file entry: /__w/3/s/target/project-0.0.1-DEV.tar
java.io.IOException: Pipe closed
        at java.base/java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:260)
        at java.base/java.io.PipedInputStream.receive(PipedInputStream.java:226)
        at java.base/java.io.PipedOutputStream.write(PipedOutputStream.java:150)
        at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:261)
        at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:210)
        at java.base/java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:148)
        at org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:62)
        at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:91)
        at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:259)
        at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:169)
        at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:206)
        at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:713)
        at java.base/java.io.InputStream.transferTo(InputStream.java:783)
        at java.base/java.nio.file.Files.copy(Files.java:3213)
...

Additional Information

No response

Start with ImageFromDockerfile and MountableFile, then trace the Docker client stream around the Broken pipe and Pipe closed errors shown in the logs. Reproduce the nested-container build with the mounted socket and verify that the target JAR can be transferred as a build context and the health-checked container starts successfully.