testcontainers / testcontainers/testcontainers-python

Bug: Reaper reports success while the session filter never reaches Ryuk (silent leak on Linux/docker-proxy)

Aperta
#1,114 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
Python
Stelle
2.3k
Fork
386
Merge medio
4h 40m
PR unite (30g)
1

Descrizione

Describe the bug

Reaper._create_instance() can return a fully "successful" reaper whose session filter never reached Ryuk. Nothing raises, Reaper._instance is set, the ryuk container is running — and yet nothing is ever reaped. When the test process is killed (CI timeout, cancelled job, OOM), every container of that session survives.

Three things combine (testcontainers/core/container.py, 4.14.2):

  1. The wait strategy is registered after the container is started, so it never applies.
Reaper._container = (
    DockerContainer(c.ryuk_image)
    ...
    .start()          # <- started here
)
rc = Reaper._container
rc.waiting_for(LogMessageWaitStrategy(r".* Started!").with_startup_timeout(20))   # <- too late

waiting_for is a builder setter — its own docstring says "Set a wait strategy to be used after container start" — so setting it on an already-started container has no effect. There is no wait for Started! in practice.

  1. docker-proxy accepts on the published port from container creation, before the ryuk process inside has bound to it. The connect loop only retries on ConnectionRefusedError/OSError, so a connection that is accepted by the proxy and then reset is indistinguishable from success:
for _ in range(50):
    try:
        s.connect((container_host, container_port))
        last_connection_exception = None
        break                      # first attempt succeeds against the proxy
    except (ConnectionRefusedError, OSError) as e:
        ...
  1. The filter is sent and the reply is never read. Ryuk answers ACK for every accepted filter line; nothing checks for it:
rs.send(f"label={LABEL_SESSION_ID}={SESSION_ID}\r\n".encode())
Reaper._instance = Reaper()

So the filter goes into a socket that is not (yet) the reaper, the peer resets it, and the library reports success.

Observed result

Ryuk's own log tells the story — no client ever registered:

Pinging Docker...
Docker daemon is available!
Starting on port 8080...
Started!
Timeout waiting for connection
Removed 0 container(s), 0 network(s), 0 volume(s), 0 image(s)

There is no New client connected line, and after the run is killed its containers stay up indefinitely. This is how a CI runner accumulated 40 leftover containers, including a full set of live storages still running hours after the run that created them had ended.

Why this is easy to miss

The failure is silent and platform-dependent. On macOS/Docker Desktop the published port does not accept connections before the process inside binds, so the library wins the race and everything looks correct. On a Linux daemon with docker-proxy it loses.

Measured: 5 out of 5 runs affected on a self-hosted Linux CI runner (Ubuntu in WSL2, Docker CE 29.1.3), 0 out of 5 on macOS with Docker Desktop, same library version and same code.

To Reproduce

On a Linux host with Docker CE:

from testcontainers.core.container import Reaper

Reaper.get_instance()
print("instance:", Reaper._instance)                  # not None — looks fine
print(Reaper._container.get_logs()[0].decode())       # no "New client connected"

# ask the socket whether anyone is on the other end
import socket
s = Reaper._socket
s.settimeout(2)
s.send(b"label=org.testcontainers.session-id=probe\r\n")
print(s.recv(64))                                     # ConnectionResetError instead of b"ACK"

Then start any container in a child process and SIGKILL the process: the container is still there a minute later, and ryuk exits with Removed 0 container(s).

Suggested fix

Two independent halves, either of which closes the hole:

  • apply the wait strategy before connecting — e.g. move waiting_for(...) above .start(), or explicitly wait for the Started! log line after starting;
  • read Ryuk's reply after sending the filter and treat a missing ACK as "not connected to the reaper", retrying the connection. This is the stronger of the two, because it verifies the property that actually matters — the reaper accepted the filter — rather than a proxy for it.

Runtime environment

  • testcontainers-python 4.14.2
  • ryuk 0.8.1 and 0.11.0 (both affected)
  • Docker CE 29.1.3 on Ubuntu (WSL2) — affected; Docker Desktop on macOS — not affected

Related but different

#1093 describes the case where s.connect(...) raises and the failure is visible. This report is the opposite: the connection succeeds, no exception is raised, and the reaper silently does nothing.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Il percorso di errore si trova in testcontainers/core/container.py, in particolare in Reaper._create_instance() e nella configurazione del socket. Inizia tracciando la registrazione della strategia di attesa e l’invio del filtro su Linux/docker-proxy, quindi riproduci il problema con Reaper.get_instance() e il controllo ACK fornito. Il lavoro è completato quando Ryuk conferma il filtro e un SIGKILL non lascia container di sessione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
docker, python
Ambito
devops, testing-qa
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Attiva
Chiarezza
Specificata chiaramente
Idoneità per principianti
68/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.