testcontainers / testcontainers/testcontainers-java

`VncRecordingContainer.reencodeRecording` hangs

Open
#5,024 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

modules/selenium
Dominant language
Java
Stars
8.7k
Forks
1.9k
Avg merge
2d 17h
Merged PRs (30d)
9

Description

Thanks for the awesome project. It really really help out. I've been experiencing this issue for several months. It happens frequently (several times per day) but I didn't find a stable way to reproduce, thus I can't provide a minimal reproducer. My project runs browser tests with Kotlin 1.6.10, Gradle 7.3. It hangs frequently if I enabled recording, regardless of format (flv/mp4). Once I disable recording, no issues at all.

Environment:

macOS 11.3.1
docker desktop 4.2.0 (70708)
testcontainers 1.16.3

When it hangs, the docker ps outputs:

$ docker ps
CONTAINER ID   IMAGE                                       COMMAND                   CREATED       STATUS       PORTS                                              NAMES
8bb3dc872278   testcontainers/vnc-recorder:1.2.0           "/bin/sh -c 'echo 'c…"    3 hours ago   Up 3 hours                                                      hungry_poincare
eff7f640ef47   selenium/standalone-chrome-debug:3.141.59   "/opt/bin/entry_poin…"    3 hours ago   Up 3 hours   0.0.0.0:64092->4444/tcp, 0.0.0.0:64093->5900/tcp   blissful_lichterman
dd764547abf5   testcontainers/vnc-recorder:1.2.0           "/bin/sh -c 'echo 'c…"    3 hours ago   Up 3 hours                                                      modest_chatterjee
b56b6c98537b   selenium/standalone-chrome-debug:3.141.59   "/opt/bin/entry_poin…"    3 hours ago   Up 3 hours   0.0.0.0:64074->4444/tcp, 0.0.0.0:64075->5900/tcp   admiring_tharp
b23acaf9723c   testcontainers/vnc-recorder:1.2.0           "/bin/sh -c 'echo 'c…"    3 hours ago   Up 3 hours                                                      cranky_jang
1692aee2351f   selenium/standalone-chrome-debug:3.141.59   "/opt/bin/entry_poin…"    3 hours ago   Up 3 hours   0.0.0.0:64054->4444/tcp, 0.0.0.0:64055->5900/tcp   heuristic_meninsky
2b2e94963b45   amazon/dynamodb-local:latest                "java -jar DynamoDBL…"    3 hours ago   Up 3 hours   0.0.0.0:64042->8000/tcp                            competent_mestorf
5e61134895bb   testcontainers/sshd:1.0.0                   "sh -c 'echo \"root:$…"   3 hours ago   Up 3 hours   0.0.0.0:63665->22/tcp                              brave_mcclintock
573d1e348e81   testcontainers/ryuk:0.3.3                   "/app"                    3 hours ago   Up 3 hours   0.0.0.0:63656->8080/tcp                            testcontainers-ryuk-cda1010c-3a9a-4725-8db4-3f4b0c00f96d

The stacktrace and jstack dump:

"Test worker" #21 prio=5 os_prio=31 cpu=9934.43ms elapsed=9287.41s tid=0x00007f98db016800 nid=0x6403 waiting on condition  [0x0000700004e9e000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)
	- parking to wait for  <0x00000007fc6b1fd0> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.11/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.11/AbstractQueuedSynchronizer.java:885)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.11/AbstractQueuedSynchronizer.java:1039)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.11/AbstractQueuedSynchronizer.java:1345)
	at java.util.concurrent.CountDownLatch.await(java.base@11.0.11/CountDownLatch.java:232)
	at com.github.dockerjava.api.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:91)
	at org.testcontainers.containers.ExecInContainerPattern.execInContainer(ExecInContainerPattern.java:80)
	at org.testcontainers.containers.ContainerState.execInContainer(ContainerState.java:235)
	at org.testcontainers.containers.ContainerState.execInContainer(ContainerState.java:226)
	at org.testcontainers.containers.VncRecordingContainer$VncRecordingFormat$2.reencodeRecording(VncRecordingContainer.java:135)
	at org.testcontainers.containers.VncRecordingContainer.streamRecording(VncRecordingContainer.java:105)
	at org.testcontainers.containers.VncRecordingContainer.saveRecordingToFile(VncRecordingContainer.java:116)
	at org.testcontainers.containers.BrowserWebDriverContainer.retainRecordingIfNeeded(BrowserWebDriverContainer.java:345)
	at org.testcontainers.containers.BrowserWebDriverContainer.afterTest(BrowserWebDriverContainer.java:303)
	at com.bytelegend.app.testfixtures.JUnit5VncRecorder.afterTestExecution(JUnit5VncRecorder.kt:61)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterTestExecutionCallbacks$9(TestMethodTestDescriptor.java:233)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$3359/0x0000000800c8ac40.invoke(Unknown Source)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:273)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$3361/0x0000000800c8fc40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$14(TestMethodTestDescriptor.java:273)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$3360/0x0000000800c8f840.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.11/ArrayList.java:1541)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:272)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterTestExecutionCallbacks(TestMethodTestDescriptor.java:232)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$248/0x00000008001e3440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$247/0x00000008001e3040.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$246/0x00000008001e2c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$252/0x00000008001e4440.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.11/ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$248/0x00000008001e3440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$247/0x00000008001e3040.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$246/0x00000008001e2c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$252/0x00000008001e4440.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@11.0.11/ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$248/0x00000008001e3440.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$247/0x00000008001e3040.invoke(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$246/0x00000008001e2c40.execute(Unknown Source)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$208/0x00000008001c8040.accept(Unknown Source)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.11/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.11/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:135)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.11/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.11/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.11/Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.11/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.11/ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)

Any help/guidance on how to mitigate would be appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at VncRecordingContainer.java:135, following reencodeRecording through streamRecording and saveRecordingToFile, and compare the supplied jstack with ExecInContainerPattern.awaitCompletion. Reproduce or isolate the intermittent Docker recording hang; done means recording re-encoding completes without leaving the test worker waiting.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
devops, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.