Bug: rootless docker-in-docker in Gitlab CI can't find containers

Describe the bug

Trying to start containers in rootless docker-in-docker running on Gitlab self-hosted runners results in docker.errors.NotFound: 404 Client Error for https://docker:2376/v1.44/containers/fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f/json: Not Found ("No such container: fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f")

To Reproduce

• Gitlab self-hosted runner
• Docker autoscaler executor
• Rootless docker-in-docker
• cp -r /certs/client/* /root/.docker because the Docker dependency doesn't respect DOCKER_TLS_CERTDIR and tcp://docker:2376 fails without certs in this dir (got this from here: https://github.com/testcontainers/testcontainers-python/blob/main/core/tests/test_docker_in_docker.py#L221)
• Run pytest that uses testcontainers

State:

CONTAINER ID   IMAGE   COMMAND   CREATED   STATUS   PORTS   NAMES
0083e6a1e9f6   47494f49e760 "dockerd-entrypoint.…"   6 minutes ago   Up 6 minutes   2375-2376/tcp   <long_container_name_ending_in>-build
60dba14e1956   4f1ab7591b83 "dockerd-entrypoint.…"   6 minutes ago   Up 6 minutes   2375-2376/tcp   <long_container_name_ending_in>__dind-rootless-0

Local:

test_everything.py::test_get_all_customers PASSED                                                                                                 [ 50%]
test_everything.py::test_get_customer_by_email PASSED                                                                                             [100%]

Remote:

`docker.errors.NotFound: 404 Client Error for https://docker:2376/v1.44/containers/fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f/json: Not Found ("No such container: fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f")`

Runtime environment

Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.

# Get the operating system information (on a unix os).
$ uname -a
# Get the python version.
$ python --version
# Get the docker version and other docker information.
$ docker info
# Get all python packages.
$ pip freeze

amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 5 17:36:57 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

python 3.11.11 in build container, requires_python = ">=3.11,<3.12" in pyproject.toml

Docker info:

Worker instance:

Client:
 Version:    25.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.0.0+unknown
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 4
  Running: 3
  Paused: 0
  Stopped: 1
 Images: 5
 Server Version: 25.0.6
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: 2c9f5602f0ba3d9da1c2596322dfc4e156844890
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.115-126.197.amzn2023.x86_64
 Operating System: Amazon Linux 2023.6.20241121
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.64GiB
 Name: ip-10-121-36-254.eu-central-1.compute.internal
 ID: 6ec60948-bbef-4c17-96de-7bee4fab9940
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Build container:

Client:
 Version:    25.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 25.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.115-126.197.amzn2023.x86_64
 Operating System: Alpine Linux v3.19
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.64GiB
 Name: 60dba14e1956
 ID: 184c448b-2a74-4591-8333-b60346a901bf
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. Systemd is required to enable cgroups in rootless-mode.

Dependencies for job:

"psycopg2-binary>=2.9.10",
"pytest>=8.3.4",
"testcontainers[postgres]>=4.8.2",

@alexanderankin Perhaps one for you given https://github.com/testcontainers/testcontainers-python/issues/517?

Start with core/tests/test_docker_in_docker.py, which covers Docker-in-Docker behavior, and reproduce the reported pytest failure using rootless Docker-in-Docker on a GitLab self-hosted runner. Trace the Docker connection and certificate setup described in the issue. Done means testcontainers can find and manage its containers in this environment without the reported 404 error.