[Bug]: HTTP connection state with podman

Module

Core

Testcontainers version

1.18.3

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

❯ docker version
Client:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Built:        Fri May 26 17:58:48 2023
OS/Arch:      linux/amd64

What happened?

We have been chasing issues between testcontainers and podman where tests would fail on MacOS with

Exception in thread "Thread-0" java.lang.RuntimeException: com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.NoHttpResponseException: localhost:2375 failed to respond

or on Linux with

java.lang.RuntimeException: java.io.IOException: Broken pipe

After much head scratching we think we have figured it out as a mismatch between the connection timeout expected by testcontainers-java and the podman API. By default the podman socket is closed after 5s however test containers expects to be able to use the connection for up to 3 mins.

I'm attaching a failing test.

Podman generally expects to be started by systemd in response to data being sent to the socket (via the podman.socket systemd unit) as it is has a demon less execution model. I'm not close enough to the details to understand why the connection fails with a broken pipe instead of re-activating the connection.

My suspicion is that we see different behaviour on MacOS as gvproxy sits in between testcontaienrs and the podman API and thus when the socket goes away it handles the broken pipe differently (possibly stalled waiting for data to come back).

Relevant log output

No response

Additional Information

Podman logs

Passes with time=100: doTheBadThing-time-100-17-07-2023_04:39.log

Fails with time=10: doTheBadThing-time-10-17-07-2023_04:44.log

Test containers debug log from the failing run

podmanTimeout.log

Testcase

Test case: DoTheBadThingTest.java.txt

Systemd customisations for changing timeouts

Podman systemd configuration override override.conf.txt which can be applied with systemctl edit --user podman.service && systemctl restart --user podman.service

Start with the Core module and the attached DoTheBadThingTest.java.txt testcase, then review the podmanTimeout.log and the passing and failing Podman logs. Reproduce the connection behavior with the stated timeout settings on Linux, and verify that the testcase no longer produces NoHttpResponseException or Broken pipe failures with Podman.