testcontainers / testcontainers/testcontainers-node

Reaper discovery adopts another language binding's Ryuk, then leaks every container (findReaperContainers matches only org.testcontainers.ryuk)

Open
#1,442 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
2.6k
Forks
272
PR merge metrics
No merged PRs in 30d

Description

Expected Behaviour

Reaper discovery should only adopt a Ryuk container that belongs to testcontainers-node, or — if it adopts one belonging to another language binding — it should register its session with that reaper so the containers it creates are still reaped.

Actual Behaviour

findReaperContainers matches any running Ryuk on the host, regardless of which language binding started it. On a CI host shared between a testcontainers-node project and a testcontainers-python project, the Node workers adopt the Python binding's Ryuk and then leak every container they create.

packages/testcontainers/src/reaper/reaper.ts (verified in the published build/reaper/reaper.js of both 11.14.0 and 12.1.0 — the code is identical in the two):

async function findReaperContainers(client) {
    const containers = await client.container.list();
    return containers
        .filter((container) => container.State === "running" &&
        container.Labels[LABEL_TESTCONTAINERS_RYUK] === "true" &&
        container.Labels["TESTCONTAINERS_RYUK_TEST_LABEL"] !== "true")
        .sort((a, b) => b.Created - a.Created);
}

The only positive predicate is the image label org.testcontainers.ryuk=true. That label is on the testcontainers/ryuk image itself, so every binding's reaper carries it — confirmed on two versions in use side by side:

$ docker image inspect testcontainers/ryuk:0.14.0 --format '{{json .Config.Labels}}'
{"org.testcontainers.ryuk":"true"}
$ docker image inspect testcontainers/ryuk:0.8.1  --format '{{json .Config.Labels}}'
{"org.testcontainers.ryuk":"true"}

There is no check on org.testcontainers.lang, on org.testcontainers.version, or on the image tag — even though this library writes org.testcontainers.lang: "node" on everything it creates (utils/labels.ts, createLabels()).

Why adoption then loses the session. A reaper started by another binding carries no org.testcontainers.session-id label, so this line mints a fresh id per worker:

const existingSessionId = reaperContainer.Labels[LABEL_TESTCONTAINERS_SESSION_ID] ?? new RandomUuid().nextUuid();

The adopted reaper is then asked to watch a session it was never told about in a form it durably owns, and the containers created under that id are never reaped.

Testcontainer Logs

From the foreign (Python-started) reaper's own log while Node workers were running against it. Its own session id is a full UUID; the 12-hex ids are the ones this library minted per worker:

Adding {"label":{"org.testcontainers.session-id=cf5e8a51-7464-4627-9ede-6c092b479713":true}}
Adding {"label":{"org.testcontainers.session-id=3297709e6c31":true}}
Adding {"label":{"org.testcontainers.session-id=57a593d04394":true}}

The observable signature is one leaked container per distinct session id, with no reaper alive that ever owned them. We saw six containers survive with six distinct session ids and no test runner process alive, then reproduced the id-per-worker pattern at 13/13 and 23/23.

Ruled out by measurement, so these are not the cause: TESTCONTAINERS_RYUK_DISABLED is unset everywhere (the reaper is running and doing its job for live sessions), and there was no OOM kill.

Steps to Reproduce
  1. On one Docker host, start a testcontainers-python suite so that its testcontainers/ryuk container is running.
  2. While it is up, run any testcontainers-node suite against the same host.
  3. Observe the Node run log Reusing existing Reaper for session "<id>" and that the adopted reaper is the Python one.
  4. Let the Node suite end. Its containers are not reaped.
Environment
  • testcontainers 11.14.0; the same code is present in 12.1.0 (latest at time of writing), so this is not fixed by upgrading.
  • Node 22.18.0, Linux, Docker with the containerd image store.
  • testcontainers/ryuk 0.14.0 (this library) and 0.8.1 (the other binding) both present.
Suggested fix

Narrow the discovery predicate so a reaper is only adopted when this binding can actually own the session. Any of:

  • require org.testcontainers.lang === "node" on the reaper container, matching what createLabels() already writes; or
  • require the reaper container to carry an org.testcontainers.session-id label at all, rather than falling back to a freshly minted id — adopting a reaper whose session cannot be identified is the step that silently breaks reaping; or
  • match on the expected REAPER_IMAGE tag.

Happy to open a PR if you would like one, and to test a candidate against the mixed-binding host where this reproduces.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read packages/testcontainers/src/reaper/reaper.ts and utils/labels.ts, starting at findReaperContainers and createLabels(); compare the discovery labels with the ownership and session behavior described in the issue. Reproduce on a shared Docker host with testcontainers-python and testcontainers-node, then verify the chosen ownership rule prevents Node containers from being left unreaped.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
devops, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.