testcontainers / testcontainers/testcontainers-java
Container.followOutput() leaks thread
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 1.9k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 9
Description
Container.followOutput() uses LogUtils.attachConsumer(), which opens a long-running okhttp request and processes it using a FrameConsumerResultCallback.
The problem is that the FrameConsumerResultCallback, so it can't be closed after stopping the container, and the server side doesn't close the connection either, so the processing thread stays alive forever.
Our test suite uses an inheritable ThreadLocal to track and report thread leaks, and this is what it reports:
23:30:59,651 ERROR [TestSuiteProgress] Test failed: UNKNOWN.ThreadLeakChecker
org.infinispan.commons.test.ThreadLeakChecker$LeakException: Leaked thread: tc-okhttp-stream-1282372255 << testng-GracefulShutdownRestartIT << UNKNOWN
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocketLibrary.read(Native Method) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket$UnixDomainSocketInputStream.doRead(UnixDomainSocket.java:149) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket$UnixDomainSocketInputStream.read(UnixDomainSocket.java:136) ~[testcontainers-1.12.4.jar:?]
at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
at org.testcontainers.dockerclient.transport.okhttp.UnixSocketFactory$1$1.read(UnixSocketFactory.java:46) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okio.Okio$2.read(Okio.java:140) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okio.RealBufferedSource.request(RealBufferedSource.java:72) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okio.RealBufferedSource.require(RealBufferedSource.java:65) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:307) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.shaded.okio.RealBufferedSource.exhausted(RealBufferedSource.java:61) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder$FramedSink.accept(OkHttpInvocationBuilder.java:363) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder$FramedSink.accept(OkHttpInvocationBuilder.java:352) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$3(OkHttpInvocationBuilder.java:314) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder$$Lambda$1863/0x0000000100fd5840.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: org.infinispan.commons.test.ThreadLeakChecker$LeakException: testng-GracefulShutdownRestartIT << UNKNOWN
at org.infinispan.commons.test.ThreadLeakChecker$ThreadInfoLocal.childValue(ThreadLeakChecker.java:107) ~[infinispan-commons-test-11.0.0-SNAPSHOT.jar:11.0.0-SNAPSHOT]
at org.infinispan.commons.test.ThreadLeakChecker$ThreadInfoLocal.childValue(ThreadLeakChecker.java:104) ~[infinispan-commons-test-11.0.0-SNAPSHOT.jar:11.0.0-SNAPSHOT]
at java.lang.ThreadLocal$ThreadLocalMap.<init>(ThreadLocal.java:411) ~[?:?]
at java.lang.ThreadLocal.createInheritedMap(ThreadLocal.java:276) ~[?:?]
at java.lang.Thread.<init>(Thread.java:450) ~[?:?]
at java.lang.Thread.<init>(Thread.java:709) ~[?:?]
at java.lang.Thread.<init>(Thread.java:630) ~[?:?]
at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.executeAndStream(OkHttpInvocationBuilder.java:319) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.executeAndStream(OkHttpInvocationBuilder.java:295) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.get(OkHttpInvocationBuilder.java:89) ~[testcontainers-1.12.4.jar:?]
at com.github.dockerjava.core.exec.LogContainerCmdExec.execute0(LogContainerCmdExec.java:42) ~[testcontainers-1.12.4.jar:?]
at com.github.dockerjava.core.exec.LogContainerCmdExec.execute0(LogContainerCmdExec.java:12) ~[testcontainers-1.12.4.jar:?]
at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.execute(AbstrAsyncDockerCmdExec.java:56) ~[testcontainers-1.12.4.jar:?]
at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.exec(AbstrAsyncDockerCmdExec.java:21) ~[testcontainers-1.12.4.jar:?]
at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec.exec(AbstrAsyncDockerCmdExec.java:12) ~[testcontainers-1.12.4.jar:?]
at com.github.dockerjava.core.command.AbstrAsyncDockerCmd.exec(AbstrAsyncDockerCmd.java:21) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.utility.LogUtils.attachConsumer(LogUtils.java:99) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.utility.LogUtils.followOutput(LogUtils.java:36) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.utility.LogUtils.followOutput(LogUtils.java:51) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.containers.Container.followOutput(Container.java:391) ~[testcontainers-1.12.4.jar:?]
at java.util.ArrayList.forEach(ArrayList.java:1540) ~[?:?]
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:412) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:317) ~[testcontainers-1.12.4.jar:?]
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81) ~[duct-tape-1.0.8.jar:?]
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:315) ~[testcontainers-1.12.4.jar:?]
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302) ~[testcontainers-1.12.4.jar:?]
at org.infinispan.server.test.ContainerInfinispanServerDriver.start(ContainerInfinispanServerDriver.java:146) ~[test-classes/:?]
at org.infinispan.server.test.InfinispanServerDriver.start(InfinispanServerDriver.java:109) ~[test-classes/:?]
at org.infinispan.server.test.InfinispanServerRule$1.evaluate(InfinispanServerRule.java:86) ~[test-classes/:?]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Container.followOutput() and LogUtils.attachConsumer(), then trace the FrameConsumerResultCallback lifecycle and the okhttp streaming request shown in the stack trace. Reproduce the leak through GracefulShutdownRestartIT and verify that stopping the container no longer leaves the tc-okhttp-stream thread running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100