[Bug]: Parallel container startup hungs when withCreateContainerCmdModifier is used

Module

Core

Testcontainers version

1.19.8

Using the latest Testcontainers version?

Yes

Host OS

Windows 11

Host Arch

x86

Docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:48:43 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:28 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Startables.deepStart(container).join(); hungs indefinitely when container has withCreateContainerCmdModifier defined.

Example:

public class AppTest {

    @SuppressWarnings("resource")
    static GenericContainer<?> gcs = new GenericContainer<>(DockerImageName.parse("fsouza/fake-gcs-server:1"))
            .withExposedPorts(4443)
            .withCreateContainerCmdModifier(cmd -> cmd.withEntrypoint("/bin/fake-gcs-server", "-scheme", "http"));

    static {
        startGlobalContainersParallel();
    }

    static void startGlobalContainersSequentially() {
        gcs.start();
    }

    static void startGlobalContainersParallel() {
        Startables.deepStart(gcs).join();
    }

    @Test
    public void shouldAnswerWithTrue() {
        assertTrue(true);
    }

}

If Startables.deepStart is not used (i.e. instead of calling startGlobalContainersParallel() we call startGlobalContainersSequentially()), then all work fine.

Removing withCreateContainerCmdModifier also helps.

I tried to add withCreateContainerCmdModifier to other containers (e.g. MySQLContainer) and it hungs too, so this bug doesn't depend on the container.

Relevant log output

19:12:53.402 [testcontainers-lifecycle-0] INFO tc.testcontainers/ryuk:0.7.0 -- Container testcontainers/ryuk:0.7.0 started in PT0.7214507S
19:12:53.406 [testcontainers-ryuk] DEBUG org.testcontainers.utility.ResourceReaper -- Sending 'label=org.testcontainers%3Dtrue&label=org.testcontainers.lang%3Djava&label=org.testcontainers.version%3D1.19.8&label=org.testcontainers.sessionId%3D2300acd2-8cc9-41ef-b3e1-37ea382ee1a8' to Ryuk
19:12:53.406 [testcontainers-ryuk] DEBUG org.testcontainers.utility.RyukResourceReaper -- Received 'ACK' from Ryuk
19:12:53.406 [testcontainers-lifecycle-0] INFO org.testcontainers.utility.RyukResourceReaper -- Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.DockerClientFactory -- Checks are enabled
19:12:53.407 [testcontainers-lifecycle-0] INFO org.testcontainers.DockerClientFactory -- Checking the system...
19:12:53.407 [testcontainers-lifecycle-0] INFO org.testcontainers.DockerClientFactory -- ✔︎ Docker server version should be at least 1.6.0
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.PrefixingImageNameSubstitutor -- No prefix is configured
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.ImageNameSubstitutor -- Did not find a substitute image for fsouza/fake-gcs-server:1 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.images.AbstractImagePullPolicy -- Using locally available and not pulling image: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] DEBUG tc.fsouza/fake-gcs-server:1 -- Starting container: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] DEBUG tc.fsouza/fake-gcs-server:1 -- Trying to start container: fsouza/fake-gcs-server:1 (attempt 1/1)
19:12:53.407 [testcontainers-lifecycle-0] DEBUG tc.fsouza/fake-gcs-server:1 -- Starting container: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] INFO tc.fsouza/fake-gcs-server:1 -- Creating container for image: fsouza/fake-gcs-server:1
19:12:53.407 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.RegistryAuthLocator -- Looking up auth config for image: fsouza/fake-gcs-server:1 at registry: https://index.docker.io/v1/
19:12:53.408 [testcontainers-lifecycle-0] DEBUG org.testcontainers.utility.RegistryAuthLocator -- No matching Auth Configs - falling back to defaultAuthConfig [null]
19:12:53.408 [testcontainers-lifecycle-0] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig -- Effective auth config [null]

Additional Information

Minimal reproduce: tc-parallel-start-bug.zip.

Docs aboutt parallel container startup: https://java.testcontainers.org/features/advanced_options/#parallel-container-startup

Reproduce the hang using the linked minimal example and the Startables.deepStart call with withCreateContainerCmdModifier; compare it with sequential startup. Trace the parallel startup path from Startables.deepStart and inspect the container-command modifier handling. Done means modified containers start in parallel without hanging and a regression test covers the reported case.