testcontainers / testcontainers/testcontainers-java

Mapped port polling on container startup causes race conditions with containers that terminate on startup

Open
#5,385 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
8.7k
Forks
1.9k
Avg merge
2d 17h
Merged PRs (30d)
9

Description

It appears that https://github.com/testcontainers/testcontainers-java/pull/4263 introduced some flakiness in a subset of our tests. In particular, we have a few tests that test failure conditions for a container. In those tests, we either override waitUntilContainerStarted to check for a special log message or use a OneShotStartupCheckStrategy to detect when the desired state has been achieved. However, after upgrading to a version of TestContainers that includes the PR above (in this case, from 1.15.3 to 1.17.1), we started occasionally getting failures like this:

...
Caused by: org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: Lambda expression in org.testcontainers.containers.GenericContainer: expected the predicate to return <true> but it returned <false> for input of ...
	at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.testcontainers.shaded.org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:86)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:691)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:708)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:450)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:340)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:338)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:326)
...

My belief is that, when it fails, the container terminates in between polling attempts, causing the until block to never succeed.
My expectation would be that the block to detect mapped ports should not throw an exception in this case.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in GenericContainer.java at tryStart, using the stack trace and the mapped-port polling introduced by PR 4263 as entry points. Reproduce a failure-condition test with a OneShotStartupCheckStrategy or an overridden waitUntilContainerStarted while the container terminates during polling. Done means startup handling no longer times out solely because the container terminates between mapped-port polling attempts.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.