Question: Accessing docker compose container services when using DinD

What are you trying to do?

We're currently trying to use DockerCompose class objects to run integration tests inside our CI env (gitlab using DinD service).
As the docker composed set of services is being spin inside a DinD service container, they're not directly accessible from the job container,

Where are you trying to do it?

from testcontainers.compose import DockerCompose
compose = DockerCompose(
         "tests/integration/compose", compose_file_name="docker-compose.yml", pull=True
     )
     compose.start()
     print("PublishedPort: "+compose.get_container(service_name="schema-registry").get_publisher().URL)

This prints out an inaccessible URL (from the CI job container standpoint)
The same code works fine on local setup.

Runtime environment

(GitLab Job CI env settings)

uname -a
echo "------"
docker info
echo "------"
poetry run python --version
echo "------"
poetry show --tree

Linux 694dc901af1e 6.5.0-1022-aws #22~22.04.1-Ubuntu SMP Fri Jun 14 16:31:00 UTC 2024 x86_64 x86_64 x86_64 inux 694dc901af1e 6.5.0-1022-aws #22~22.04.1-Ubuntu SMP Fri Jun 14 16:31:00 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
------
Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.0.0+unknown
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
------
Path /dp-gitlab-migration-test/mms_receiver/external_modules/application/aws_logging for aws_logging does not exist
Skipping virtualenv creation, as specified in config file.
Python 3.11.6
------
Path /dp-gitlab-migration-test/mms_receiver/external_modules/application/aws_logging for aws_logging does not exist
Skipping virtualenv creation, as specified in config file.
aws-lambda-powertools 2.35.1 Powertools for AWS Lambda (Python) is a developer toolkit to implement Serverless best practices and increase developer velocity.
└── typing-extensions >=4.6.2,<5.0.0
aws-logging 1.0.0 A package that contain a reusable module for structure logging
├── boto3 ^1.17.98
│   ├── botocore >=1.34.35,<1.35.0
│   │   ├── jmespath >=0.7.1,<2.0.0
│   │   ├── python-dateutil >=2.1,<3.0.0
│   │   │   └── six >=1.5
│   │   └── urllib3 >=1.25.4,<2.1
│   ├── jmespath >=0.7.1,<2.0.0 (circular dependency aborted here)
│   └── s3transfer >=0.10.0,<0.11.0
│       └── botocore >=1.33.2,<2.0a.0 (circular dependency aborted here)
└── structlog ^22.3.0
black 23.3.0 The uncompromising code formatter.
├── click >=8.0.0
│   └── colorama *
├── mypy-extensions >=0.4.3
├── packaging >=22.0
├── pathspec >=0.9.0
└── platformdirs >=2
boto3 1.34.35 The AWS SDK for Python
├── botocore >=1.34.35,<1.35.0
│   ├── jmespath >=0.7.1,<2.0.0
│   ├── python-dateutil >=2.1,<3.0.0
│   │   └── six >=1.5
│   └── urllib3 >=1.25.4,<2.1
├── jmespath >=0.7.1,<2.0.0
└── s3transfer >=0.10.0,<0.11.0
    └── botocore >=1.33.2,<2.0a.0
        ├── jmespath >=0.7.1,<2.0.0
        ├── python-dateutil >=2.1,<3.0.0
        │   └── six >=1.5
        └── urllib3 >=1.25.4,<2.1
confluent-kafka 2.3.0 Confluent's Python client for Apache Kafka
fastavro 1.9.4 Fast read/write of AVRO files
polyfactory 2.15.0 Mock data generation factories
├── faker *
│   └── python-dateutil >=2.4
│       └── six >=1.5
└── typing-extensions >=4.6.0
psycopg 3.1.18 PostgreSQL database adapter for Python
├── typing-extensions >=4.1
└── tzdata *
psycopg-binary 3.1.18 PostgreSQL database adapter for Python -- C optimisation distribution
psycopg2-binary 2.9.9 psycopg2 - Python-PostgreSQL Database Adapter
pydantic 2.6.4 Data validation using Python type hints
├── annotated-types >=0.4.0
├── email-validator >=2.0.0
│   ├── dnspython >=2.0.0
│   └── idna >=2.0.0
├── pydantic-core 2.16.3
│   └── typing-extensions >=4.6.0,<4.7.0 || >4.7.0
└── typing-extensions >=4.6.1
pytest 7.2.2 pytest: simple powerful testing with Python
├── attrs >=19.2.0
├── colorama *
├── iniconfig *
├── packaging *
└── pluggy >=0.12,<2.0
pytest-cov 4.0.0 Pytest plugin for measuring coverage.
├── coverage >=5.2.1
└── pytest >=4.6
    ├── attrs >=19.2.0
    ├── colorama *
    ├── iniconfig *
    ├── packaging *
    └── pluggy >=0.12,<2.0
requests 2.31.0 Python HTTP for Humans.
├── certifi >=2017.4.17
├── charset-normalizer >=2,<4
├── idna >=2.5,<4
└── urllib3 >=1.21.1,<3
sqlalchemy 2.0.28 Database Abstraction Library
├── greenlet !=0.4.17
└── typing-extensions >=4.6.0
testcontainers 4.5.1 Python library for throwaway instances of anything that can run in a Docker container
├── docker *
│   ├── pywin32 >=304
│   ├── requests >=2.26.0
│   │   ├── certifi >=2017.4.17
│   │   ├── charset-normalizer >=2,<4
│   │   ├── idna >=2.5,<4
│   │   └── urllib3 >=1.21.1,<3
│   └── urllib3 >=1.26.0 (circular dependency aborted here)
├── typing-extensions *
├── urllib3 *
└── wrapt *
tox 4.4.8 tox is a generic virtualenv management and test command line tool
├── cachetools >=5.3
├── chardet >=5.1
├── colorama >=0.4.6
├── filelock >=3.10
├── packaging >=23
├── platformdirs >=3.1.1
├── pluggy >=1
├── pyproject-api >=1.5.1
│   └── packaging >=23.1
└── virtualenv >=20.21
    ├── distlib >=0.3.7,<1
    ├── filelock >=3.12.2,<4
    └── platformdirs >=3.9.1,<5

Start with the DockerCompose flow shown in tests/integration/compose, especially compose.start(), get_container(), and get_publisher(). Reproduce the example in the GitLab DinD environment and compare the published URL with local behavior. Done means establishing whether the URL can be made reachable from the job container or documenting the required supported configuration.