[Bug]: HTTP connection state with podman

Module

Core

Testcontainers version

1.18.3

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

❯ docker version
Client:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Built:        Fri May 26 17:58:48 2023
OS/Arch:      linux/amd64

What happened?

We have been chasing issues between testcontainers and podman where tests would fail on MacOS with

Exception in thread "Thread-0" java.lang.RuntimeException: com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.NoHttpResponseException: localhost:2375 failed to respond

or on Linux with

java.lang.RuntimeException: java.io.IOException: Broken pipe

After much head scratching we think we have figured it out as a mismatch between the connection timeout expected by testcontainers-java and the podman API. By default the podman socket is closed after 5s however test containers expects to be able to use the connection for up to 3 mins.

I'm attaching a failing test.

Podman generally expects to be started by systemd in response to data being sent to the socket (via the podman.socket systemd unit) as it is has a demon less execution model. I'm not close enough to the details to understand why the connection fails with a broken pipe instead of re-activating the connection.

My suspicion is that we see different behaviour on MacOS as gvproxy sits in between testcontaienrs and the podman API and thus when the socket goes away it handles the broken pipe differently (possibly stalled waiting for data to come back).

Relevant log output

No response

Additional Information

Podman logs

Passes with time=100: doTheBadThing-time-100-17-07-2023_04:39.log

Fails with time=10: doTheBadThing-time-10-17-07-2023_04:44.log

Test containers debug log from the failing run

podmanTimeout.log

Testcase

Test case: DoTheBadThingTest.java.txt

Systemd customisations for changing timeouts

Podman systemd configuration override override.conf.txt which can be applied with systemctl edit --user podman.service && systemctl restart --user podman.service

Beginne mit dem Core module und dem angehängten DoTheBadThingTest.java.txt testcase und überprüfe anschließend podmanTimeout.log sowie die erfolgreichen und fehlgeschlagenen Podman logs. Reproduziere das Verbindungsverhalten mit den angegebenen Timeout-Einstellungen unter Linux und verifiziere, dass das testcase mit Podman keine NoHttpResponseException- oder Broken pipe-Fehler mehr erzeugt.