Bug: rootless docker-in-docker in Gitlab CI can't find containers

Describe the bug

Trying to start containers in rootless docker-in-docker running on Gitlab self-hosted runners results in docker.errors.NotFound: 404 Client Error for https://docker:2376/v1.44/containers/fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f/json: Not Found ("No such container: fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f")

To Reproduce

• Gitlab self-hosted runner
• Docker autoscaler executor
• Rootless docker-in-docker
• cp -r /certs/client/* /root/.docker because the Docker dependency doesn't respect DOCKER_TLS_CERTDIR and tcp://docker:2376 fails without certs in this dir (got this from here: https://github.com/testcontainers/testcontainers-python/blob/main/core/tests/test_docker_in_docker.py#L221)
• Run pytest that uses testcontainers

State:

CONTAINER ID   IMAGE   COMMAND   CREATED   STATUS   PORTS   NAMES
0083e6a1e9f6   47494f49e760 "dockerd-entrypoint.…"   6 minutes ago   Up 6 minutes   2375-2376/tcp   <long_container_name_ending_in>-build
60dba14e1956   4f1ab7591b83 "dockerd-entrypoint.…"   6 minutes ago   Up 6 minutes   2375-2376/tcp   <long_container_name_ending_in>__dind-rootless-0

Local:

test_everything.py::test_get_all_customers PASSED                                                                                                 [ 50%]
test_everything.py::test_get_customer_by_email PASSED                                                                                             [100%]

Remote:

`docker.errors.NotFound: 404 Client Error for https://docker:2376/v1.44/containers/fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f/json: Not Found ("No such container: fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f")`

Runtime environment

Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.

# Get the operating system information (on a unix os).
$ uname -a
# Get the python version.
$ python --version
# Get the docker version and other docker information.
$ docker info
# Get all python packages.
$ pip freeze

amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 5 17:36:57 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

python 3.11.11 in build container, requires_python = ">=3.11,<3.12" in pyproject.toml

Docker info:

Worker instance:

Client:
 Version:    25.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.0.0+unknown
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 4
  Running: 3
  Paused: 0
  Stopped: 1
 Images: 5
 Server Version: 25.0.6
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: 2c9f5602f0ba3d9da1c2596322dfc4e156844890
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.115-126.197.amzn2023.x86_64
 Operating System: Amazon Linux 2023.6.20241121
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.64GiB
 Name: ip-10-121-36-254.eu-central-1.compute.internal
 ID: 6ec60948-bbef-4c17-96de-7bee4fab9940
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Build container:

Client:
 Version:    25.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 25.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.115-126.197.amzn2023.x86_64
 Operating System: Alpine Linux v3.19
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.64GiB
 Name: 60dba14e1956
 ID: 184c448b-2a74-4591-8333-b60346a901bf
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. Systemd is required to enable cgroups in rootless-mode.

Dependencies for job:

"psycopg2-binary>=2.9.10",
"pytest>=8.3.4",
"testcontainers[postgres]>=4.8.2",

@alexanderankin Perhaps one for you given https://github.com/testcontainers/testcontainers-python/issues/517?

Commencez par core/tests/test_docker_in_docker.py, qui couvre le comportement de Docker-in-Docker, et reproduisez l’échec pytest signalé avec Docker-in-Docker rootless sur un runner GitLab auto-hébergé. Suivez la configuration de la connexion Docker et des certificats décrite dans l’issue. Le travail est terminé lorsque testcontainers peut trouver et gérer ses conteneurs dans cet environnement sans l’erreur 404 signalée.