testcontainers / testcontainers/testcontainers-python

Bug: Reaper reports success while the session filter never reaches Ryuk (silent leak on Linux/docker-proxy)

Offen
#1,114 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Vorherrschende Sprache
Python
Sterne
2.3k
Forks
386
Ø Merge
4 Std. 40 Min.
Gemergte PRs (30 T.)
1

Beschreibung

Describe the bug

Reaper._create_instance() can return a fully "successful" reaper whose session filter never reached Ryuk. Nothing raises, Reaper._instance is set, the ryuk container is running — and yet nothing is ever reaped. When the test process is killed (CI timeout, cancelled job, OOM), every container of that session survives.

Three things combine (testcontainers/core/container.py, 4.14.2):

  1. The wait strategy is registered after the container is started, so it never applies.
Reaper._container = (
    DockerContainer(c.ryuk_image)
    ...
    .start()          # <- started here
)
rc = Reaper._container
rc.waiting_for(LogMessageWaitStrategy(r".* Started!").with_startup_timeout(20))   # <- too late

waiting_for is a builder setter — its own docstring says "Set a wait strategy to be used after container start" — so setting it on an already-started container has no effect. There is no wait for Started! in practice.

  1. docker-proxy accepts on the published port from container creation, before the ryuk process inside has bound to it. The connect loop only retries on ConnectionRefusedError/OSError, so a connection that is accepted by the proxy and then reset is indistinguishable from success:
for _ in range(50):
    try:
        s.connect((container_host, container_port))
        last_connection_exception = None
        break                      # first attempt succeeds against the proxy
    except (ConnectionRefusedError, OSError) as e:
        ...
  1. The filter is sent and the reply is never read. Ryuk answers ACK for every accepted filter line; nothing checks for it:
rs.send(f"label={LABEL_SESSION_ID}={SESSION_ID}\r\n".encode())
Reaper._instance = Reaper()

So the filter goes into a socket that is not (yet) the reaper, the peer resets it, and the library reports success.

Observed result

Ryuk's own log tells the story — no client ever registered:

Pinging Docker...
Docker daemon is available!
Starting on port 8080...
Started!
Timeout waiting for connection
Removed 0 container(s), 0 network(s), 0 volume(s), 0 image(s)

There is no New client connected line, and after the run is killed its containers stay up indefinitely. This is how a CI runner accumulated 40 leftover containers, including a full set of live storages still running hours after the run that created them had ended.

Why this is easy to miss

The failure is silent and platform-dependent. On macOS/Docker Desktop the published port does not accept connections before the process inside binds, so the library wins the race and everything looks correct. On a Linux daemon with docker-proxy it loses.

Measured: 5 out of 5 runs affected on a self-hosted Linux CI runner (Ubuntu in WSL2, Docker CE 29.1.3), 0 out of 5 on macOS with Docker Desktop, same library version and same code.

To Reproduce

On a Linux host with Docker CE:

from testcontainers.core.container import Reaper

Reaper.get_instance()
print("instance:", Reaper._instance)                  # not None — looks fine
print(Reaper._container.get_logs()[0].decode())       # no "New client connected"

# ask the socket whether anyone is on the other end
import socket
s = Reaper._socket
s.settimeout(2)
s.send(b"label=org.testcontainers.session-id=probe\r\n")
print(s.recv(64))                                     # ConnectionResetError instead of b"ACK"

Then start any container in a child process and SIGKILL the process: the container is still there a minute later, and ryuk exits with Removed 0 container(s).

Suggested fix

Two independent halves, either of which closes the hole:

  • apply the wait strategy before connecting — e.g. move waiting_for(...) above .start(), or explicitly wait for the Started! log line after starting;
  • read Ryuk's reply after sending the filter and treat a missing ACK as "not connected to the reaper", retrying the connection. This is the stronger of the two, because it verifies the property that actually matters — the reaper accepted the filter — rather than a proxy for it.

Runtime environment

  • testcontainers-python 4.14.2
  • ryuk 0.8.1 and 0.11.0 (both affected)
  • Docker CE 29.1.3 on Ubuntu (WSL2) — affected; Docker Desktop on macOS — not affected

Related but different

#1093 describes the case where s.connect(...) raises and the failure is visible. This report is the opposite: the connection succeeds, no exception is raised, and the reaper silently does nothing.

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Der Fehlerpfad befindet sich in testcontainers/core/container.py, insbesondere in Reaper._create_instance() und beim Socket-Setup. Beginne damit, die Registrierung der Wait-Strategie und das Senden des Filters unter Linux/docker-proxy nachzuverfolgen, und reproduziere das Problem mit Reaper.get_instance() und der bereitgestellten ACK-Prüfung. Als abgeschlossen gilt die Aufgabe, wenn Ryuk den Filter bestätigt und ein SIGKILL keine Session-Container hinterlässt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
docker, python
Bereich
devops, testing-qa
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
68/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.