Bug: `ConnectionRefusedError: [Errno 111] Connection refused` via docker-compose on Github Actions

Describe the bug

This seems potentially related to https://github.com/testcontainers/testcontainers-python/issues/517 and https://github.com/testcontainers/testcontainers-python/issues/475

TLDR; I am able to connect to the PG testcontainer from within docker running locally on a macbook, but running in github actions I get a connection error.

I'm experiencing a ConnectionRefusedError: [Errno 111] Connection refused error when attempting connecting to the postgres testcontainer when running my tests via docker-compose on a Github action. However I am able to run the tests within docker-compose locally on my macbook.

I followed the instructions here for running testcontainers from within docker

https://golang.testcontainers.org/system_requirements/ci/gitlab_ci/#example-using-docker-socket

This is the error I see in Github Actions

        for _ in range(50):
            try:
                Reaper._socket = socket()
>               Reaper._socket.connect((container_host, container_port))
E               ConnectionRefusedError: [Errno 111] Connection refused
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:228: ConnectionRefusedError

Below is the full stack trace

________________________ ERROR at setup of test_search _________________________
    @pytest.fixture(scope="session")
    def db_url():
        print("YO")
        print(set([a[4][0] for a in socket.getaddrinfo(socket.gethostname(), None)]))
>       with PostgresContainer("postgres:13.1") as postgres:
tests/conftest.py:21: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:111: in __enter__
    return self.start()
/venv/lib/python3.11/site-packages/testcontainers/core/generic.py:70: in start
    super().start()
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:87: in start
    Reaper.get_instance()
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:188: in get_instance
    Reaper._instance = Reaper._create_instance()
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:242: in _create_instance
    raise last_connection_exception
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cls = <class 'testcontainers.core.container.Reaper'>
    @classmethod
    def _create_instance(cls) -> "Reaper":
        logger.debug(f"Creating new Reaper for session: {SESSION_ID}")
    
        Reaper._container = (
            DockerContainer(c.ryuk_image)
            .with_name(f"testcontainers-ryuk-{SESSION_ID}")
            .with_exposed_ports(8080)
            .with_volume_mapping(c.ryuk_docker_socket, "/var/run/docker.sock", "rw")
            .with_kwargs(privileged=c.ryuk_privileged, auto_remove=True)
            .with_env("RYUK_RECONNECTION_TIMEOUT", c.ryuk_reconnection_timeout)
            .start()
        )
        wait_for_logs(Reaper._container, r".* Started!")
    
        container_host = Reaper._container.get_container_host_ip()
        container_port = int(Reaper._container.get_exposed_port(8080))
    
        last_connection_exception: Optional[Exception] = None
        for _ in range(50):
            try:
                Reaper._socket = socket()
>               Reaper._socket.connect((container_host, container_port))
E               ConnectionRefusedError: [Errno 111] Connection refused
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:228: ConnectionRefusedError

The following is my test setup:

conftest.py

@pytest.fixture(scope="session")
def db_url():
    with PostgresContainer("postgres:13.1") as postgres:
        url = postgres.get_connection_url()
        yield url

The following is the docker command I'm using to run the tests via make test. I've also tried multiple variations of the volume mapping to with no luck.

test:
	docker-compose run --rm \
	 -v ${PWD}:${PWD} -w ${PWD}/server \
	 -v /var/run/docker.sock:/var/run/docker.sock \
	 --entrypoint "" server poetry run pytest -vv

where the server image is in my docker-compose.yml is

  server:
    container_name: svs_server
    platform: linux/amd64
    build: ./server
    image: "svs_server:local"
    ports:
      - "8000:8000"
    volumes:
      - "./server:/app"
      - "./db:/app/db"
      - "./tmp:/tmp"
    env_file:
      - .env
      - .env.secret

I am able to successfully get the testcontainers/docker-compose setup running locally on my macbook (docker desktop) with the following, notice the only difference is the additional host override and lack of mapped volumes.

test-local:
	docker-compose run --rm \
	 -v /var/run/docker.sock:/var/run/docker.sock \
	 -e TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal \
	 --entrypoint "" server poetry run pytest -vv

Runtime environment

I also tried finding more info about the network via running the following within the action.

print(set([a[4][0] for a in socket.getaddrinfo(socket.gethostname(), None)]))

This returns an IP such as {'172.18.0.2'} but it is slightly different each time.

• Running within Github Actions
• Python 3.11 via the FROM python:3.11-slim-buster image
• docker info

Client: Docker Engine - Community Version: 24.0.9 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.14.0 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.23.3 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 14 Server Version: 24.0.9 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: e377cd56a71523140ca6ae87e30244719194a521 runc version: v1.1.12-0-g51d5e94 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.5.0-1018-azure Operating System: Ubuntu 22.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 7.744GiB Name: fv-az822-516 ID: 584d62f5-11d9-4074-87e4-acacf60905a6 Docker Root Dir: /var/lib/docker Debug Mode: false Username: githubactions Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

• docker-compose version
  docker-compose version 1.29.2, build 5becea4c

• uname -a
Linux fv-az822-516 6.5.0-1018-azure #19~22.04.2-Ubuntu SMP Thu Mar 21 16:45:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Let me know if there's any other additional debugging info I can provide that would help, thanks!!

Start with tests/conftest.py and the Reaper._create_instance path shown in the traceback, then compare the Makefile test command and docker-compose.yml between GitHub Actions and the working local command. Reproduce the failure with the mounted Docker socket and inspect the host override and container networking. Done means the PostgreSQL testcontainer setup runs successfully in GitHub Actions or the required configuration and limitation are documented.