Fatal error happened occasionaly(multi-threads)
Personne n'a encore pris cette issue.
- Langage dominant
- Java
- Étoiles
- 928
- Forks
- 227
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 18.04
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- TensorFlow installed from (source or binary):source
- TensorFlow version (use command below):TF Java 0.2.0
- Python version:python3.7
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:
You can collect some of this information using our environment capture script
You can also obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the current behavior
some fatal error happend occasionaly when I test the code in multiple threads:
Describe the expected behavior
How to avoid this situation? This fatal error killed the entire Java service
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
1.code:
`// get tensorflow serving response
List servingResultList = responseFaceDet.getOutputsMap().get(concat).getFloatValList();
Float[] servingResultArray = servingResultList.toArray(new Float[15232]);
FloatDataBuffer floatDataBuffer = NioDataBufferFactory.create(FloatBuffer.allocate(15232));
floatDataBuffer.write(servingResultArray);
FloatNdArray servingResultNdArray = NdArrays.wrap(Shape.of(896, 17), floatDataBuffer);
float[][] servingBoxResultArray = StdArrays.array2dCopyOf(servingResultNdArray);
// prepare data for non max suppression
FloatNdArray scores1dArray = servingResultNdArray.slice(all(), at(16));
FloatNdArray a = servingResultNdArray.slice(all(), seq(1, 0));
FloatNdArray b = servingResultNdArray.slice(all(), seq(3, 2));
//generate constant max filled 2
FloatDataBuffer constantArrayBuffer = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
Float[] fConst2 = new Float[1796];
Arrays.fill(fConst2, 2.f);
constantArrayBuffer.write(fConst2);
FloatNdArray constantArray = NdArrays.wrap(Shape.of(896, 2), constantArrayBuffer);
//generate constant max filled 0
FloatDataBuffer constantArrayBuffer0 = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
Float[] fConst0 = new Float[1796];
Arrays.fill(fConst0, 0.f);
constantArrayBuffer0.write(fConst0);
FloatNdArray constantArray0 = NdArrays.wrap(Shape.of(896, 2), constantArrayBuffer0);
//generate constant max filled max value
FloatDataBuffer constantArrayBufferNull = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
Float[] fConstNull = new Float[1796];
Arrays.fill(fConstNull, 1230f);
constantArrayBufferNull.write(fConstNull);
FloatNdArray constantArrayNull = NdArrays.wrap(Shape.of(896, 2), constantArrayBufferNull);
try (EagerSession session = EagerSession.create()) {
Ops tf = Ops.create(session);
Div<TFloat32> div = tf.math.div(tf.constant(b), tf.constant(constantArray));
Neg<TFloat32> neg = tf.math.neg(div);
Add<TFloat32> add = tf.math.add(neg, tf.constant(a));
Add<TFloat32> add2 = tf.math.add(div, tf.constant(a));
ClipByValue<TFloat32> clipByValue = tf.clipByValue(add, tf.constant(constantArray0), tf.constant(constantArrayNull));
Concat<TFloat32> finalBoxes = tf.concat(Arrays.asList(clipByValue, add2), tf.constant(1));
NonMaxSuppression nonMaxSuppression = tf.image.nonMaxSuppression(finalBoxes, tf.constant(scores1dArray), tf.constant(5), tf.constant(0.5f), tf.constant(FACE_THRESHOLD), tf.constant(0.01f));
Output<TInt32> indiceOutput = nonMaxSuppression.selectedIndices();
int[] indexNMS = StdArrays.array1dCopyOf(indiceOutput.data());
Output<TFloat32> scoreOutput = nonMaxSuppression.selectedScores();
float[] scoreNMS = StdArrays.array1dCopyOf(scoreOutput.data());
List scoreList = new ArrayList();
float[] boxArray;
if (scoreNMS.length > 0) {
boxArray = servingBoxResultArray[indexNMS[0]];
scoreList.add(scoreNMS[0]);
log.info("scoreList index 0 = {}", scoreNMS[0]);
} else {
boxArray = new float[4];
}
float offsetXOri = (float) dx / (float) FACEDET_SIZE_NEW;
float offsetYOri = (float) dy / (float) FACEDET_SIZE_NEW;
float scaleXOri = (float) FACEDET_SIZE_NEW / (float) faceDetWidth;
float scaleYOri = (float) FACEDET_SIZE_NEW / (float) faceDetHeight;
// get Top1 highest score Box
centX = (int) ((boxArray[0] - offsetXOri) * scaleYOri * finalHeight);
centY = (int) ((boxArray[1] - offsetYOri) * scaleXOri * finalWidth);
responseWidth = (int) (boxArray[2] * scaleYOri * finalHeight);
responseHeight = (int) (boxArray[3] * scaleXOri * finalWidth);
CvRect cvRect2 = new CvRect();
cvRect2.x((centX - responseWidth / 2));
cvRect2.y((centY - responseHeight / 2));
cvRect2.width(responseWidth);
cvRect2.height(responseHeight);
Position topLeft = Position.newBuilder().setX((centX - responseWidth / 2)).setY((centY - responseHeight / 2)).build();
Position bottomRight = Position.newBuilder().setX((centX + responseWidth / 2)).setY((centY + responseHeight / 2)).build();
facePosition.add(topLeft);
facePosition.add(bottomRight);
} catch (StatusRuntimeException e) {
if (e.getStatus().getCode() == Status.Code.DEADLINE_EXCEEDED) {
log.warn("detectMotion face model postprocessing failed!!!", e);
return ModelCheckResponse.newBuilder()
.setFace(true)
.build();
} else {
throw e;
}
}`
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
2.hs_err_pid24993.log
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007f0d1d772691, pid=13893, tid=0x00007f0c76afa700
JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
Problematic frame:
C [libc.so.6+0x16f691] __strlen_sse2_pminub+0x11
Core dump written. Default location: /opt/app/logs/bd-pica-api/core or core.13893
If you would like to submit a bug report, please visit:
http://bugreport.java.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.
--------------- T H R E A D ---------------
Current thread (0x00007f0c88389800): JavaThread "grpc-executor-48" daemon [_thread_in_native, id=22083, stack(0x00007f0c769fa000,0x00007f0c76afb000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f0b502e32e0
Registers:
RAX=0x0000000000000000, RBX=0x00007f0c76af9198, RCX=0x0000000000000020, RDX=0x00007f0b57ff8900
RSP=0x00007f0c76af9168, RBP=0x00007f0c76af91e0, RSI=0x0000000000000000, RDI=0x00007f0b502e32e0
R8 =0x00007f0c8c63c290, R9 =0x0000000000000002, R10=0x0000000000000000, R11=0x00007f0d1d5d6f88
R12=0x00007f0c76af91f8, R13=0x00007f0b502e32e0, R14=0x00007f0c8c63c110, R15=0x00007f0bd3c49510
RIP=0x00007f0d1d772691, EFLAGS=0x0000000000010287, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f0c76af9168)
0x00007f0c76af9168: 00007f0c4ecc0153 0000000000000002
0x00007f0c76af9178: 00007f0c8c63c290 00007f0b57ff8900
0x00007f0c76af9188: 00007f0c8ffbb540 00000007c02be468
0x00007f0c76af9198: 0000000000000000 0000000000000000
0x00007f0c76af91a8: 0000000000000000 00000007bf985ad8
0x00007f0c76af91b8: 00000007bf985b00 00007f0c8c683c00
0x00007f0c76af91c8: 00000007bf985b5c 00007f0c76af9250
0x00007f0c76af91d8: 00007f0c88389800 00007f0c76af9210
0x00007f0c76af91e8: 00007f0c4ec38138 00007f0c76af9260
0x00007f0c76af91f8: 0000000000000000 00000007bf985b00
0x00007f0c76af9208: 0000000000000000 00007f0c76af9260
0x00007f0c76af9218: 00007f0d09122d4d 00000007bf985720
0x00007f0c76af9228: 00000007bf985c10 00000007bf985b68
0x00007f0c76af9238: 00000007bf985cc0 0000001a48257a79
0x00007f0c76af9248: 0000000000000003 00000006c0baf2e8
0x00007f0c76af9258: 00007f0c5593e4b7 00000007bf985cf0
0x00007f0c76af9268: 00007f0d097f3964 00000007bf985cc0
0x00007f0c76af9278: 00000007bf9789e8 00000007bf985ad8
0x00007f0c76af9288: 00000007bf985720 00000007bf985b68
0x00007f0c76af9298: 00000007bf985c10 00000007bf985cc0
0x00007f0c76af92a8: 00007f0c4ece5f7f 00000007bf9859f8
0x00007f0c76af92b8: 00000007bf985990 00000006c0baf2e8
0x00007f0c76af92c8: 00000007bf9859f4 00000000f7f30b45
0x00007f0c76af92d8: 00007f0d08ec31a0 00000007bf985a28
0x00007f0c76af92e8: 0000000000000000 00000007bf985ac0
0x00007f0c76af92f8: 00007f0d09b2d4ac 00000000f7f30b45
0x00007f0c76af9308: 00007f0d091e8278 00000007bf985968
0x00007f0c76af9318: 00000007bf985570 00000007bf985a58
0x00007f0c76af9328: 00000007bf985980 0000001a48257a79
0x00007f0c76af9338: 0000000000000003 00000006c0baf2e8
0x00007f0c76af9348: 00000007bf96dd28 00000007bf985ac0
0x00007f0c76af9358: 00007f0d0933aa6c 00000007bf985ac0
Instructions: (pc=0x00007f0d1d772691)
0x00007f0d1d772671: c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48
0x00007f0d1d772681: 31 c0 89 f9 83 e1 3f 66 0f ef c0 83 f9 30 77 1d
0x00007f0d1d772691: f3 0f 6f 0f 66 0f 74 c1 66 0f d7 d0 85 d2 0f 85
0x00007f0d1d7726a1: 4e 02 00 00 48 89 f8 48 83 e0 f0 eb 24 48 89 f8
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value
RBX=0x00007f0c76af9198 is pointing into the stack for thread: 0x00007f0c88389800
RCX=0x0000000000000020 is an unknown value
RDX=0x00007f0b57ff8900 is an unknown value
RSP=0x00007f0c76af9168 is pointing into the stack for thread: 0x00007f0c88389800
RBP=0x00007f0c76af91e0 is pointing into the stack for thread: 0x00007f0c88389800
RSI=0x0000000000000000 is an unknown value
RDI=0x00007f0b502e32e0 is an unknown value
R8 =0x00007f0c8c63c290 is an unknown value
R9 =0x0000000000000002 is an unknown value
R10=0x0000000000000000 is an unknown value
R11=0x00007f0d1d5d6f88: <offset 0xfc3f88> in /opt/programs/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so at 0x00007f0d1c613000
R12=0x00007f0c76af91f8 is pointing into the stack for thread: 0x00007f0c88389800
R13=0x00007f0b502e32e0 is an unknown value
R14=0x00007f0c8c63c110 is an unknown value
R15=0x00007f0bd3c49510 is an unknown value
Stack: [0x00007f0c769fa000,0x00007f0c76afb000], sp=0x00007f0c76af9168, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x16f691] __strlen_sse2_pminub+0x11
C [libtensorflow_cc.so.2+0x2385138] TFE_Execute+0x28
J 3768 org.tensorflow.internal.c_api.global.tensorflow.TFE_Execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/bytedeco/javacpp/PointerPointer;Lorg/bytedeco/javacpp/IntPointer;Lorg/tensorflow/internal/c_api/TF_Status;)V (0 bytes) @ 0x00007f0d09122d4d [0x00007f0d09122c40+0x10d]
J 5826 C2 org.tensorflow.EagerOperationBuilder.execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/tensorflow/EagerSession;)[Lorg/tensorflow/internal/c_api/TFE_TensorHandle; (210 bytes) @ 0x00007f0d097f3964 [0x00007f0d097f36e0+0x284]
J 6675 C2 org.tensorflow.EagerOperationBuilder.build()Lorg/tensorflow/EagerOperation; (57 bytes) @ 0x00007f0d09b2d4ac [0x00007f0d09b2d460+0x4c]
J 8643 C2 org.tensorflow.op.MathOps.div(Lorg/tensorflow/Operand;Lorg/tensorflow/Operand;)Lorg/tensorflow/op/math/Div; (10 bytes) @ 0x00007f0d0933aa6c [0x00007f0d09339e80+0xbec]
J 7481 C2 com.em.api.algo.impl.FaceProcessorSerivceImpl.getModelInfo(Lcom/em/api/face/ModelCheckRequest;)Lcom/em/api/face/ModelCheckResponse; (2376 bytes) @ 0x00007f0d09f98068 [0x00007f0d09f94140+0x3f28]
J 8623 C2 com.em.api.face.ModelCheckServiceGrpc$MethodHandlers.invoke(Ljava/lang/Object;Lio/grpc/stub/StreamObserver;)V (48 bytes) @ 0x00007f0d0987d630 [0x00007f0d0987d000+0x630]
J 8652 C2 io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext()V (73 bytes) @ 0x00007f0d08afc0f4 [0x00007f0d08afc020+0xd4]
J 7467 C2 io.grpc.internal.ContextRunnable.run()V (35 bytes) @ 0x00007f0d09e77828 [0x00007f0d09e77620+0x208]
J 7931 C2 io.grpc.internal.SerializingExecutor.run()V (99 bytes) @ 0x00007f0d0904b930 [0x00007f0d0904b6a0+0x290]
J 9207 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f0d096d3994 [0x00007f0d096d36e0+0x2b4]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x690206] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x690711] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V [libjvm.so+0x690bb7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V [libjvm.so+0x7278f0] thread_entry(JavaThread*, Thread*)+0xa0
V [libjvm.so+0xa742b3] JavaThread::thread_main_inner()+0x103
V [libjvm.so+0xa743fc] JavaThread::run()+0x11c
V [libjvm.so+0x924ea8] java_start(Thread*)+0x108
C [libpthread.so.0+0x7ea5] start_thread+0xc5
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 3768 org.tensorflow.internal.c_api.global.tensorflow.TFE_Execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/bytedeco/javacpp/PointerPointer;Lorg/bytedeco/javacpp/IntPointer;Lorg/tensorflow/internal/c_api/TF_Status;)V (0 bytes) @ 0x00007f0d09122ccb [0x00007f0d09122c40+0x8b]
J 5826 C2 org.tensorflow.EagerOperationBuilder.execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/tensorflow/EagerSession;)[Lorg/tensorflow/internal/c_api/TFE_TensorHandle; (210 bytes) @ 0x00007f0d097f3964 [0x00007f0d097f36e0+0x284]
J 6675 C2 org.tensorflow.EagerOperationBuilder.build()Lorg/tensorflow/EagerOperation; (57 bytes) @ 0x00007f0d09b2d4ac [0x00007f0d09b2d460+0x4c]
J 8643 C2 org.tensorflow.op.MathOps.div(Lorg/tensorflow/Operand;Lorg/tensorflow/Operand;)Lorg/tensorflow/op/math/Div; (10 bytes) @ 0x00007f0d0933aa6c [0x00007f0d09339e80+0xbec]
J 7481 C2 com.em.api.algo.impl.FaceProcessorSerivceImpl.getModelInfo(Lcom/em/api/face/ModelCheckRequest;)Lcom/em/api/face/ModelCheckResponse; (2376 bytes) @ 0x00007f0d09f98068 [0x00007f0d09f94140+0x3f28]
J 8623 C2 com.em.api.face.ModelCheckServiceGrpc$MethodHandlers.invoke(Ljava/lang/Object;Lio/grpc/stub/StreamObserver;)V (48 bytes) @ 0x00007f0d0987d630 [0x00007f0d0987d000+0x630]
J 8652 C2 io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext()V (73 bytes) @ 0x00007f0d08afc0f4 [0x00007f0d08afc020+0xd4]
J 7467 C2 io.grpc.internal.ContextRunnable.run()V (35 bytes) @ 0x00007f0d09e77828 [0x00007f0d09e77620+0x208]
J 7931 C2 io.grpc.internal.SerializingExecutor.run()V (99 bytes) @ 0x00007f0d0904b930 [0x00007f0d0904b6a0+0x290]
J 9207 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f0d096d3994 [0x00007f0d096d36e0+0x2b4]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
Guide de contribution
Ouvrir le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Piste de recherche
Commencez par le code Java multithread fourni et le fichier hs_err_pid24993.log, en vous concentrant sur la frame native TFE_Execute et le SIGSEGV intermittent. Établissez un test minimal reproductible pour TF Java 0.2.0 sur Ubuntu 18.04 ; le travail est considéré comme terminé lorsque le crash natif est expliqué et qu’un contournement ou une correction vérifiée est identifié.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- backend
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 30/100