Can't connect to PostgreSQL container from another container when running on Mac as host

In our current setup we have some Python code running on a docker container which uses testcontainers to create a PostgreSQL database container we can test with. All necessary requirements to start a new container from a docker image are met, and the host machine's docker client is used to start the new container. When looking at the DB container, it starts up successfully, we can access it from localhost with the port assigned to it when exposing and with the container's IP with the port we wanted to expose (5432). The problem comes if we try to access it with the Gateway IP of the network the 2 containers are in, and the port assigned to it when exposing (which is what testcontainers is doing as per my understanding).

This last part only fails when the host the 2 containers are started on is a MacOS machine, it works fine on Linux. To replicate the issue the following script can be run on a docker container that is started on a MacOS machine (with the host's docker client mounted up as per the requirements in this repo's README):

from testcontainers.postgres import PostgresContainer
import sqlalchemy

with PostgresContainer("postgres:9.6") as postgres:
    e = sqlalchemy.create_engine(postgres.get_connection_url())
    print(e.execute("select version()"))

We ran with Python 3.6 and newest versions of testcontainers, sqlalchemy and psycopg2. The above script should fail in a container started on Mac and work in a container started on Linux.

This is because the networking implementation is different between Linux and Mac. When exposing ports on Mac without providing a binding port (so docker finds a port for itself), the HostConfig/PortBindings for the newly started container gets set to "0". In contrast, it is left as an empty String on Linux. The "0" value causes the issue. We've opened a ticket about this on the docker for mac repo which has more info about this particular issue: https://github.com/docker/for-mac/issues/5588

Currently the work around we use in our code is to explicitly bind ports instead of just exposing them like so:

from testcontainers.postgres import PostgresContainer
import sqlalchemy

with PostgresContainer("postgres:9.6").with_bind_ports(5432, 47000) as postgres:
    e = sqlalchemy.create_engine(postgres.get_connection_url())
    print(e.execute("select version()"))

This version works on both Mac and Linux as the host because HostConfig/PortBindings gets updated with the port we've provided on containers started on both Mac and Linux. The downside is we always have to manually bind and find a port that is available.

The suggestion in the issue that I linked above is that for portability, the internal DNS and IP of containers should be used for communicating between containers rather than the Gateway IP. Would it make sense to make the change in testcontainers so it would use those to access the DB container rather than the Gateway IP? What was the specific reason for which Gateway IP was chosen as the preferred method of communication between containers?

I am happy to have a go at making the change in this repo if you accept contributions.

Start with the PostgresContainer entry points get_connection_url and with_bind_ports, then run the supplied SQLAlchemy reproduction in containers on Mac and Linux. Trace how the database container address and exposed port are selected; done means the query connects without manually choosing a host port and behaves consistently on both hosts.