Could not connect to Ryuk during running java tests via a docker container.

まず ResourceReaper.start とスタックトレースに記載されている DockerClientFactory のパスから始め、Docker 20.10.8 と testcontainers/ryuk:0.3.3 を使って Maven コンテナのセットアップを再現します。固定ポートとランダムポートの Ryuk マッピングを比較し、Maven コンテナが報告されたゲートウェイと動的に割り当てられたポートを介して接続できることを確認します。

Hi All

We are running our tests within a maven-java container during our gitlab pipeline. Recently we had to migrate to the newest docker version (since gitlab runner got upgraded. ) and we are getting the following connection issue between the two containers (java-maven and ryuk).

java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:56144 at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:227) at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:219) at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:101) at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:107) at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:316) at TestMariaDB.mariaDBContainer(TestMariaDB.java:14) at TestMariaDB.name(TestMariaDB.java:20) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

With some trials I was able to reproduce it locally (after I ugraded to docker 20.x ; I must say all worked find with docker 19.x but not an option anymore on our gitlabserver to stay on 19.x).

To give you an idea some snippets we do and can be reproduced.

docker run -v ~/work:/build 3.8.3-jdk-11 mvn test -f /build/sample/pom.xml
I am using on my mac (but fails also on linux gitlab runner)

 docker -v
Docker version 20.10.8, build 3967b7d

I was investigation what goes wrong and found:

When from 'maven-container' wants to contact ryuk via the gateway 172.17.0.1:56144 the connection cannot be established. But I could curl the ryuk container myself using the container's specific ip and 8080 port.
curl 172.18.0.2:8080 ACK ACK ACK ACK

Checking the code of test-containers I've seen its using the

public final class ResourceReaper {
.....
public static String start(DockerClient client) {
....

   String host = containerState.getHost();
        Integer ryukPort = containerState.getFirstMappedPort();
.....

Now I started wondering and playing around what goes wrong between the two versions of docker.

If I start ryuk manually with -p 1234:8080 I can easily curl ryuk from mvn-java container

Docker inpsect:

 "HostConfig": {
      "Binds": [
        "/var/run/docker.sock:/var/run/docker.sock"
      ],
      "ContainerIDFile": "",
      "LogConfig": {
        "Type": "json-file",
        "Config": {}
      },
      "NetworkMode": "default",
      "PortBindings": {
        "8080/tcp": [
          {
            "HostIp": "",
            **"HostPort": "1234"**
          }
        ]
      },

docker run -v /var/run/docker.sock:/var/run/docker.sock -p 1234:8080 testcontainers/ryuk:0.3.3
curl 172.17.0.1:56144 ACK ACK ACK ACK

Now the problem comes if I start ryuk with a random port which will be the case what is done via test-containers.

docker run -v /var/run/docker.sock:/var/run/docker.sock -p 8080 testcontainers/ryuk:0.3.3
`
Docker inspect

"HostConfig": {
           "Binds": [
               "/var/run/docker.sock:/var/run/docker.sock"
           ],
           "ContainerIDFile": "",
           "LogConfig": {
               "Type": "json-file",
               "Config": {}
           },
           "NetworkMode": "default",
           "PortBindings": {
               **"8080/tcp": [
                   {
                       "HostIp": "0.0.0.0",
                       "HostPort": "0"
                   }**
               ]
           },
...............
NetworkSettings": {
           "Bridge": "",
           "SandboxID": "546e2822939191384c59a4c4686f677f0137fdead848adaa1c22548c2dfee782",
           "HairpinMode": false,
           "LinkLocalIPv6Address": "",
           "LinkLocalIPv6PrefixLen": 0,
           "Ports": {
               "8080/tcp": [
                   {
                       "HostIp": "0.0.0.0",
                       "HostPort": "57226"
                   }
               ]
           },
           "SandboxKey": "/var/run/docker/netns/546e28229391",
           "SecondaryIPAddresses": null,
           "SecondaryIPv6Addresses": null,
           "EndpointID": "d553bcc6bf272dac330f90315caacbb5b77673f64bf1ca34cd80dd01e9dd8480",
           "Gateway": "172.17.0.1",
           "GlobalIPv6Address": "",
           "GlobalIPv6PrefixLen": 0,
           "IPAddress": "172.17.0.2",

Notice that the HostIp its 0.0.0.0 and HostPort: 0 on HostConfig. Having this kind of setup with 'random' port start I could not
curl anymore the ryuk (with newer docker) from my maven container curl 172.17.0.1:57226 only its container ip curl 172.17.0.2:8080.

I suspect that there is something has to do with the way the newer docker is handling the random ports.

"PortBindings": {
                **"8080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "0"
                    }**
                ]

I forced a port addition during ryuk container creation and that did the trick ryuk was reachable. As well when the newer docker engine created the ryuk container. Actually it filled the HostPort to "HostPort": "1234";

String ryukContainerId = client.createContainerCmd(ryukImage)
                .withHostConfig(
                    new HostConfig()
                        .withAutoRemove(true)
                        .withPortBindings(new PortBinding(**Ports.Binding.bindPort(1234),** ryukExposedPort))
                )
                .withExposedPorts(ryukExposedPort)
                .withName("testcontainers-ryuk-" + DockerClientFactory.SESSION_ID)
                .withLabels(Collections.singletonMap(DockerClientFactory.TESTCONTAINERS_LABEL, "true"))
                .withBinds(binds)
                .withPrivileged(TestcontainersConfiguration.getInstance().isRyukPrivileged())
                .exec()
                .getId(); 

Sure we could work it around by disabling ryuk and doe some work around for our mariadb container like assining a port or getting the container IP. But perhaps there is some settings / fixes which can be done fixing this.
Any thoughts ?

Thanks
Alpar