testcontainers / testcontainers/testcontainers-java

[Bug]: Can't connect to Ryuk when running in container on Jenkins

Open
#8,577 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
Java
Stars
8.7k
Forks
1.9k
Avg merge
2d 17h
Merged PRs (30d)
9

Description

Module

Core

Testcontainers version

1.19.7

Using the latest Testcontainers version?

Yes

Host OS

CentOS Linux 7 (Core)

Host Arch

Linux 3.10.0-327.36.1.el7.x86_64

Docker version
Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:39:02 2023
 OS/Arch:           linux/amd64
 Context:           default
What happened?

I'm Trying to run some tests with TestContainers inside a Jenkins pipeline, with those two command :
docker build -f Dockerfile-package-application -t build-container . --progress=plain --network=host

docker run -e TESTCONTAINERS_HOST_OVERRIDE=localhost --network=host -v /etc/passwd:/etc/passwd:ro --rm -v ${playlist_services_directory}:${playlist_services_directory} -w ${playlist_services_directory} -v /var/run/docker.sock:/var/run/docker.sock -t build-container

and the dockerfile has a basic CMD ["mvn", "clean", "install" ]

--network=host is for allowing the containers to upload the maven dependancies via a Mirror

The launch of the test in Intellij or in a docker (In Macbook pro M1) is very smooth and get the result I want, but not on Jenkins, where I receive those Logs, where ryuk tries a couple of times before it crashes. Note that I used different strategies like -e TESTCONTAINERS_HOST_OVERRIDE=host or -e TESTCONTAINERS_HOST_OVERRIDE=172.17.0.1 but the issue remains only on Jenkins.
it a question of networking a believe and it's a real Struggle.

Relevant log output
12:59:26.992 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy -- Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
12:59:27.009 [main] INFO org.testcontainers.DockerClientFactory -- Docker host IP address is localhost
12:59:27.029 [main] INFO org.testcontainers.DockerClientFactory -- Connected to docker: 
  Server Version: 24.0.5
  API Version: 1.43
  Operating System: CentOS Linux 7 (Core)
  Total Memory: 29982 MB
12:59:27.061 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling docker image: testcontainers/ryuk:0.6.0. Please be patient; this may take some time but only needs to be done once.
12:59:27.066 [main] INFO org.testcontainers.utility.RegistryAuthLocator -- Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:latest, configFile: /root/.docker/config.json, configEnv: DOCKER_AUTH_CONFIG). Falling back to docker-java default behaviour. Exception message: Status 404: No config supplied. Checked in order: /root/.docker/config.json (file not found), DOCKER_AUTH_CONFIG (not set)
12:59:28.038 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Starting to pull image
12:59:28.062 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling image layers:  0 pending,  0 downloaded,  0 extracted, (0 bytes/0 bytes)
12:59:28.647 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling image layers:  2 pending,  1 downloaded,  0 extracted, (311 KB/? MB)
12:59:28.717 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling image layers:  1 pending,  2 downloaded,  0 extracted, (3 MB/? MB)
12:59:28.723 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling image layers:  0 pending,  3 downloaded,  0 extracted, (3 MB/6 MB)
12:59:29.194 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling image layers:  0 pending,  3 downloaded,  1 extracted, (3 MB/6 MB)
12:59:29.439 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling image layers:  0 pending,  3 downloaded,  2 extracted, (3 MB/6 MB)
12:59:29.687 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pulling image layers:  0 pending,  3 downloaded,  3 extracted, (6 MB/6 MB)
12:59:29.708 [docker-java-stream--623331358] INFO tc.testcontainers/ryuk:0.6.0 -- Pull complete. 3 layers, pulled in 1s (downloaded 6 MB at 6 MB/s)
12:59:29.708 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Image testcontainers/ryuk:0.6.0 pull took PT2.646862225S
12:59:29.730 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Creating container for image: testcontainers/ryuk:0.6.0
12:59:30.028 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Container testcontainers/ryuk:0.6.0 is starting: bfe8f38ba2a40ee6fcb1766a5625e2b73882a6b1fdeb8c07db4100cea9d5ff1d
12:59:30.362 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Container testcontainers/ryuk:0.6.0 started in PT0.631759785S
12:59:33.372 [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper -- Can not connect to Ryuk at localhost:32797
java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:318)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:350)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:393)
	at java.base/sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:217)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:171)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:188)
	at java.base/java.io.BufferedReader.fill(BufferedReader.java:160)
	at java.base/java.io.BufferedReader.implReadLine(BufferedReader.java:370)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:347)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:436)
	at org.testcontainers.utility.ResourceReaper$FilterRegistry.waitForAcknowledgment(ResourceReaper.java:397)
	at org.testcontainers.utility.ResourceReaper$FilterRegistry.register(ResourceReaper.java:393)
	at org.testcontainers.utility.RyukResourceReaper.lambda$null$1(RyukResourceReaper.java:122)
	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
	at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$2(RyukResourceReaper.java:101)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Additional Information

Note : I'm not Able to reproduce this behaviour locally, my build and tests with the same command works perfectly
No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the RyukResourceReaper stack trace and the Docker networking options in the reported Jenkins commands. Reproduce the failure in the Jenkins container setup, then compare the localhost and host-override connection paths shown in the logs. Done means Ryuk starts and accepts the resource-registration connection reliably in that environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
devops, networking, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.