testcontainers / testcontainers/testcontainers-python
Bug: Reaper reports success while the session filter never reaches Ryuk (silent leak on Linux/docker-proxy)
Chưa có ai nhận issue này.
- Ngôn ngữ chính
- Python
- Star
- 2.3k
- Fork
- 386
- Merge trung bình
- 4 giờ 40 phút
- Pull request đã merge (30 ngày)
- 1
Mô tả
Describe the bug
Reaper._create_instance() can return a fully "successful" reaper whose session filter never reached Ryuk. Nothing raises, Reaper._instance is set, the ryuk container is running — and yet nothing is ever reaped. When the test process is killed (CI timeout, cancelled job, OOM), every container of that session survives.
Three things combine (testcontainers/core/container.py, 4.14.2):
- The wait strategy is registered after the container is started, so it never applies.
Reaper._container = (
DockerContainer(c.ryuk_image)
...
.start() # <- started here
)
rc = Reaper._container
rc.waiting_for(LogMessageWaitStrategy(r".* Started!").with_startup_timeout(20)) # <- too late
waiting_for is a builder setter — its own docstring says "Set a wait strategy to be used after container start" — so setting it on an already-started container has no effect. There is no wait for Started! in practice.
docker-proxyaccepts on the published port from container creation, before the ryuk process inside has bound to it. The connect loop only retries onConnectionRefusedError/OSError, so a connection that is accepted by the proxy and then reset is indistinguishable from success:
for _ in range(50):
try:
s.connect((container_host, container_port))
last_connection_exception = None
break # first attempt succeeds against the proxy
except (ConnectionRefusedError, OSError) as e:
...
- The filter is sent and the reply is never read. Ryuk answers
ACKfor every accepted filter line; nothing checks for it:
rs.send(f"label={LABEL_SESSION_ID}={SESSION_ID}\r\n".encode())
Reaper._instance = Reaper()
So the filter goes into a socket that is not (yet) the reaper, the peer resets it, and the library reports success.
Observed result
Ryuk's own log tells the story — no client ever registered:
Pinging Docker...
Docker daemon is available!
Starting on port 8080...
Started!
Timeout waiting for connection
Removed 0 container(s), 0 network(s), 0 volume(s), 0 image(s)
There is no New client connected line, and after the run is killed its containers stay up indefinitely. This is how a CI runner accumulated 40 leftover containers, including a full set of live storages still running hours after the run that created them had ended.
Why this is easy to miss
The failure is silent and platform-dependent. On macOS/Docker Desktop the published port does not accept connections before the process inside binds, so the library wins the race and everything looks correct. On a Linux daemon with docker-proxy it loses.
Measured: 5 out of 5 runs affected on a self-hosted Linux CI runner (Ubuntu in WSL2, Docker CE 29.1.3), 0 out of 5 on macOS with Docker Desktop, same library version and same code.
To Reproduce
On a Linux host with Docker CE:
from testcontainers.core.container import Reaper
Reaper.get_instance()
print("instance:", Reaper._instance) # not None — looks fine
print(Reaper._container.get_logs()[0].decode()) # no "New client connected"
# ask the socket whether anyone is on the other end
import socket
s = Reaper._socket
s.settimeout(2)
s.send(b"label=org.testcontainers.session-id=probe\r\n")
print(s.recv(64)) # ConnectionResetError instead of b"ACK"
Then start any container in a child process and SIGKILL the process: the container is still there a minute later, and ryuk exits with Removed 0 container(s).
Suggested fix
Two independent halves, either of which closes the hole:
- apply the wait strategy before connecting — e.g. move
waiting_for(...)above.start(), or explicitly wait for theStarted!log line after starting; - read Ryuk's reply after sending the filter and treat a missing
ACKas "not connected to the reaper", retrying the connection. This is the stronger of the two, because it verifies the property that actually matters — the reaper accepted the filter — rather than a proxy for it.
Runtime environment
- testcontainers-python 4.14.2
- ryuk 0.8.1 and 0.11.0 (both affected)
- Docker CE 29.1.3 on Ubuntu (WSL2) — affected; Docker Desktop on macOS — not affected
Related but different
#1093 describes the case where s.connect(...) raises and the failure is visible. This report is the opposite: the connection succeeds, no exception is raised, and the reaper silently does nothing.
Hướng dẫn đóng góp
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Hướng nghiên cứu
Đường dẫn gây lỗi nằm trong testcontainers/core/container.py, đặc biệt là Reaper._create_instance() và phần thiết lập socket. Hãy bắt đầu bằng cách theo dõi việc đăng ký wait-strategy và việc gửi bộ lọc trên Linux/docker-proxy, sau đó tái hiện bằng Reaper.get_instance() và kiểm tra ACK được cung cấp. Được xem là hoàn tất khi Ryuk xác nhận bộ lọc và một SIGKILL không để lại session container nào.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- docker, python
- Lĩnh vực
- devops, testing-qa
- Loại issue
- Lỗi
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 68/100