temporalio / temporalio/sdk-java

Deadlock Detector unexpectedly firing on a cold JVM

Open
#1,141 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
433
Forks
249
Avg merge
5d 6h
Merged PRs (30d)
26

Description

Cold JVM is very vulnerable to JavaSDK Potential Deadlock fiting. Below there is a stacktrace with PotentialDeadlockException fires on workflow-root thread which doesn't have any user code at all and it fails on classloading and instrumentation.

JavaSDK should implement a relaxed Deadlock Detector during the start of workers.
The most straightforward solution is to multiple the Deadlock detector by N for the first execution of a workflow of each type.

Caused by: io.temporal.internal.sync.PotentialDeadlockException: Potential deadlock detected: workflow thread "workflow-root" didn't yield control for over a second. Other workflow threads:
​
​
​
	at app//org.jacoco.agent.rt.internal_3570298.asm.ByteVector.enlarge(ByteVector.java:357)
	at app//org.jacoco.agent.rt.internal_3570298.asm.ByteVector.putByte(ByteVector.java:77)
	at app//org.jacoco.agent.rt.internal_3570298.asm.SymbolTable.addConstantUtf8(SymbolTable.java:774)
	at app//org.jacoco.agent.rt.internal_3570298.asm.SymbolTable.addConstantNameAndType(SymbolTable.java:740)
	at app//org.jacoco.agent.rt.internal_3570298.asm.SymbolTable.addConstantMemberReference(SymbolTable.java:572)
	at app//org.jacoco.agent.rt.internal_3570298.asm.SymbolTable.addConstantMethodref(SymbolTable.java:542)
	at app//org.jacoco.agent.rt.internal_3570298.asm.MethodWriter.visitMethodInsn(MethodWriter.java:1035)
	at app//org.jacoco.agent.rt.internal_3570298.core.internal.instr.DuplicateFrameEliminator.visitMethodInsn(DuplicateFrameEliminator.java:78)
	at app//org.jacoco.agent.rt.internal_3570298.core.internal.instr.ClassFieldProbeArrayStrategy.storeInstance(ClassFieldProbeArrayStrategy.java:55)
	at app//org.jacoco.agent.rt.internal_3570298.core.internal.instr.ProbeInserter.visitCode(ProbeInserter.java:96)
	at app//org.jacoco.agent.rt.internal_3570298.asm.MethodVisitor.visitCode(MethodVisitor.java:232)
	at app//org.jacoco.agent.rt.internal_3570298.asm.MethodVisitor.visitCode(MethodVisitor.java:232)
	at app//org.jacoco.agent.rt.internal_3570298.asm.MethodVisitor.visitCode(MethodVisitor.java:232)
	at app//org.jacoco.agent.rt.internal_3570298.asm.tree.MethodNode.accept(MethodNode.java:742)
	at app//org.jacoco.agent.rt.internal_3570298.core.internal.flow.MethodProbesVisitor.accept(MethodProbesVisitor.java:165)
	at app//org.jacoco.agent.rt.internal_3570298.core.internal.flow.ClassProbesAdapter$2.visitEnd(ClassProbesAdapter.java:89)
	at app//org.jacoco.agent.rt.internal_3570298.asm.ClassReader.readMethod(ClassReader.java:1495)
	at app//org.jacoco.agent.rt.internal_3570298.asm.ClassReader.accept(ClassReader.java:721)
	at app//org.jacoco.agent.rt.internal_3570298.asm.ClassReader.accept(ClassReader.java:401)
	at app//org.jacoco.agent.rt.internal_3570298.core.instr.Instrumenter.instrument(Instrumenter.java:90)
	at app//org.jacoco.agent.rt.internal_3570298.core.instr.Instrumenter.instrument(Instrumenter.java:108)
	at app//org.jacoco.agent.rt.internal_3570298.CoverageTransformer.transform(CoverageTransformer.java:92)
	at java.instrument@13.0.2/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
	at java.instrument@13.0.2/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument@13.0.2/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base@13.0.2/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base@13.0.2/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
	at java.base@13.0.2/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:151)
	at java.base@13.0.2/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:821)
	at java.base@13.0.2/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719)
	at java.base@13.0.2/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:642)
	at java.base@13.0.2/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:600)
	at java.base@13.0.2/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base@13.0.2/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.MemberDeserializer.loadProperty(MemberDeserializer.kt:92)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.computeProperties(DeserializedMemberScope.kt:314)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.access$computeProperties(DeserializedMemberScope.kt:228)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation$properties$1.invoke(DeserializedMemberScope.kt:253)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation$properties$1.invoke(DeserializedMemberScope.kt:253)
	at app//kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$MapBasedMemoizedFunction.invoke(LockBasedStorageManager.java:578)
	at app//kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$MapBasedMemoizedFunctionToNotNull.invoke(LockBasedStorageManager.java:651)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.getContributedVariables(DeserializedMemberScope.kt:338)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.addFunctionsAndPropertiesTo(DeserializedMemberScope.kt:352)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope.computeDescriptors(DeserializedMemberScope.kt:115)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope$allDescriptors$1.invoke(DeserializedClassDescriptor.kt:257)
	at app//kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope$allDescriptors$1.invoke(DeserializedClassDescriptor.kt:256)
	at app//kotlin.reflect.jvm.internal.impl.storage.Loc...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the JavaSDK deadlock detector and worker startup path described in the issue, then trace the workflow-root execution and the first workflow of each type. Reproduce the cold-JVM scenario with the reported PotentialDeadlockException and Jacoco instrumentation stack; done means startup no longer triggers a false detector failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.