DockerComposeContainer: random failures with SocatContainer or ContainerisedDockerCompose

Versions

• testcontainers 1.7.3
• jdk1.8.0_172
• docker 18.03.1-ce-mac65
• docker-compose 1.21.1

Problems

• SocatContainer randomly fails to start. Using alpine/socat:1.0.3 with 5 minutes timeout for DockerComposeContainer#ambassadorContainer and withLocalCompose(true):

Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:214)
	at org.testcontainers.containers.DockerComposeContainer.startAmbassadorContainers(DockerComposeContainer.java:221)
	at org.testcontainers.containers.DockerComposeContainer.starting(DockerComposeContainer.java:131)
	at utils.TeamCityInDocker$Companion$instance$2.invoke(TeamCityInDocker.kt:59)
	at utils.TeamCityInDocker$Companion$instance$2.invoke(TeamCityInDocker.kt:36)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at utils.TeamCityInDocker$Companion.getInstance(TeamCityInDocker.kt)
	... 47 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:83)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:207)
	... 53 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:279)
	at org.testcontainers.containers.GenericContainer.lambda$start$0(GenericContainer.java:209)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
	... 54 more
Caused by: org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:51)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilTrue(Unreliables.java:95)
	at org.testcontainers.containers.startupcheck.StartupCheckStrategy.waitUntilStartupSuccessful(StartupCheckStrategy.java:26)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:251)
	... 56 more
Caused by: java.lang.RuntimeException: Not ready yet
	at org.rnorth.ducttape.unreliables.Unreliables.lambda$retryUntilTrue$1(Unreliables.java:97)
	at org.rnorth.ducttape.unreliables.Unreliables.lambda$retryUntilSuccess$0(Unreliables.java:41)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

• ContainerisedDockerCompose randomly (but very frequently) fails to start. Using docker/compose:1.21.2 and withLocalCompose(false):

INFO 🐳 [docker/compose:1.21.2] - Pulling docker image: docker/compose:1.21.2. Please be patient; this may take some time but only needs to be done once.
INFO 🐳 [docker/compose:1.21.2] - Creating container for image: docker/compose:1.21.2
INFO 🐳 [docker/compose:1.21.2] - Starting container with ID: 0d2413d84cd32005eadfb9168a9181aa1f994d87ae633697a43e0b4e1e677901
INFO 🐳 [docker/compose:1.21.2] - Container docker/compose:1.21.2 is starting: 0d2413d84cd32005eadfb9168a9181aa1f994d87ae633697a43e0b4e1e677901
ERROR c.g.dockerjava.core.async.ResultCallbackTemplate - Error during callback
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:214)
	at org.testcontainers.containers.ContainerisedDockerCompose.invoke(DockerComposeContainer.java:529)
	at org.testcontainers.containers.DockerComposeContainer.runWithCompose(DockerComposeContainer.java:189)
	at org.testcontainers.containers.DockerComposeContainer.pullImages(DockerComposeContainer.java:137)
	at org.testcontainers.containers.DockerComposeContainer.starting(DockerComposeContainer.java:127)
	at utils.TeamCityInDocker$Companion$instance$2.invoke(TeamCityInDocker.kt:59)
	at utils.TeamCityInDocker$Companion$instance$2.invoke(TeamCityInDocker.kt:36)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at utils.TeamCityInDocker$Companion.getInstance(TeamCityInDocker.kt)
	... 22 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:83)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:207)
	... 30 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:279)
	at org.testcontainers.containers.GenericContainer.lambda$start$0(GenericContainer.java:209)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
	... 31 more
Caused by: com.github.dockerjava.api.exception.NotFoundException: {"message":"No such container: 0d2413d84cd32005eadfb9168a9181aa1f994d87ae633697a43e0b4e1e677901"}

	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:103)
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
	at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
	at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934)
	at org.testcontainers.shaded.io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:593)
	at org.testcontainers.shaded.io.netty.channel.kqueue.KQueueDomainSocketChannel$KQueueDomainUnsafe.readReady(KQueueDomainSocketChannel.java:131)
	at org.testcontainers.shaded.io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:367)
	at org.testcontainers.shaded.io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:198)
	at org.testcontainers.shaded.io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:270)
	at org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

Exception gets thrown either in DockerComposeContainer#pullImages or in DockerComposeContainer#createServices. Debugging showed me that after executing CreateContainerCmd compose container just doesn't get created sometimes, docker container ps --all shows nothing.

まず DockerComposeContainer#pullImages、DockerComposeContainer#createServices、ContainerisedDockerCompose を読み、その後、それらの経路と SocatContainer の起動失敗を比較します。報告にあるバージョンと設定を使って、断続的に発生するコンテナー作成の問題を再現します。両方の compose モードが、報告されているコンテナー不足またはタイムアウトの失敗なしに、安定して起動すれば完了です。