testcontainers / testcontainers/testcontainers-java

Container.followOutput() leaks thread

Open
#2,276 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

resolution/acknowledged
Dominant language
Java
Stars
8.7k
Forks
1.9k
Avg merge
2d 17h
Merged PRs (30d)
9

Description

Container.followOutput() uses LogUtils.attachConsumer(), which opens a long-running okhttp request and processes it using a FrameConsumerResultCallback.

The problem is that the FrameConsumerResultCallback, so it can't be closed after stopping the container, and the server side doesn't close the connection either, so the processing thread stays alive forever.

Our test suite uses an inheritable ThreadLocal to track and report thread leaks, and this is what it reports:

23:30:59,651 ERROR [TestSuiteProgress] Test failed: UNKNOWN.ThreadLeakChecker
org.infinispan.commons.test.ThreadLeakChecker$LeakException: Leaked thread: tc-okhttp-stream-1282372255 << testng-GracefulShutdownRestartIT << UNKNOWN
	at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocketLibrary.read(Native Method) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket$UnixDomainSocketInputStream.doRead(UnixDomainSocket.java:149) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket$UnixDomainSocketInputStream.read(UnixDomainSocket.java:136) ~[testcontainers-1.12.4.jar:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
	at org.testcontainers.dockerclient.transport.okhttp.UnixSocketFactory$1$1.read(UnixSocketFactory.java:46) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okio.Okio$2.read(Okio.java:140) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okio.RealBufferedSource.request(RealBufferedSource.java:72) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okio.RealBufferedSource.require(RealBufferedSource.java:65) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:307) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.shaded.okio.RealBufferedSource.exhausted(RealBufferedSource.java:61) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder$FramedSink.accept(OkHttpInvocationBuilder.java:363) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder$FramedSink.accept(OkHttpInvocationBuilder.java:352) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$3(OkHttpInvocationBuilder.java:314) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder$$Lambda$1863/0x0000000100fd5840.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: org.infinispan.commons.test.ThreadLeakChecker$LeakException: testng-GracefulShutdownRestartIT << UNKNOWN
	at org.infinispan.commons.test.ThreadLeakChecker$ThreadInfoLocal.childValue(ThreadLeakChecker.java:107) ~[infinispan-commons-test-11.0.0-SNAPSHOT.jar:11.0.0-SNAPSHOT]
	at org.infinispan.commons.test.ThreadLeakChecker$ThreadInfoLocal.childValue(ThreadLeakChecker.java:104) ~[infinispan-commons-test-11.0.0-SNAPSHOT.jar:11.0.0-SNAPSHOT]
	at java.lang.ThreadLocal$ThreadLocalMap.<init>(ThreadLocal.java:411) ~[?:?]
	at java.lang.ThreadLocal.createInheritedMap(ThreadLocal.java:276) ~[?:?]
	at java.lang.Thread.<init>(Thread.java:450) ~[?:?]
	at java.lang.Thread.<init>(Thread.java:709) ~[?:?]
	at java.lang.Thread.<init>(Thread.java:630) ~[?:?]
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.executeAndStream(OkHttpInvocationBuilder.java:319) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.executeAndStream(OkHttpInvocationBuilder.java:295) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.get(OkHttpInvocationBuilder.java:89) ~[testcontainers-1.12.4.jar:?]
	at com.github.dockerjava.core.exec.LogContainerCmdExec.execute0(LogContainerCmdExec.java:42) ~[testcontainers-1.12.4.jar:?]
	at com.github.dockerjava.core.exec.LogContainerCmdExec.execute0(LogContainerCmdExec.java:12) ~[testcontainers-1.12.4.jar:?]
	at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.execute(AbstrAsyncDockerCmdExec.java:56) ~[testcontainers-1.12.4.jar:?]
	at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.exec(AbstrAsyncDockerCmdExec.java:21) ~[testcontainers-1.12.4.jar:?]
	at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.exec(AbstrAsyncDockerCmdExec.java:12) ~[testcontainers-1.12.4.jar:?]
	at com.github.dockerjava.core.command.AbstrAsyncDockerCmd.exec(AbstrAsyncDockerCmd.java:21) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.utility.LogUtils.attachConsumer(LogUtils.java:99) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.utility.LogUtils.followOutput(LogUtils.java:36) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.utility.LogUtils.followOutput(LogUtils.java:51) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.containers.Container.followOutput(Container.java:391) ~[testcontainers-1.12.4.jar:?]
	at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:412) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:317) ~[testcontainers-1.12.4.jar:?]
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81) ~[duct-tape-1.0.8.jar:?]
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:315) ~[testcontainers-1.12.4.jar:?]
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302) ~[testcontainers-1.12.4.jar:?]
	at org.infinispan.server.test.ContainerInfinispanServerDriver.start(ContainerInfinispanServerDriver.java:146) ~[test-classes/:?]
	at org.infinispan.server.test.InfinispanServerDriver.start(InfinispanServerDriver.java:109) ~[test-classes/:?]
	at org.infinispan.server.test.InfinispanServerRule$1.evaluate(InfinispanServerRule.java:86) ~[test-classes/:?]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Container.followOutput() and LogUtils.attachConsumer(), then trace the FrameConsumerResultCallback lifecycle and the okhttp streaming request shown in the stack trace. Reproduce the leak through GracefulShutdownRestartIT and verify that stopping the container no longer leaves the tc-okhttp-stream thread running.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.