tensorflow / tensorflow/java

Fatal error happened occasionaly(multi-threads)

未关闭
#326 12 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

主要语言
Java
星标
928
派生
227
PR 合并指标
30 天内没有已合并 PR

描述

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 18.04
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary):source
  • TensorFlow version (use command below):TF Java 0.2.0
  • Python version:python3.7
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version:
  • GPU model and memory:

You can collect some of this information using our environment capture script
You can also obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the current behavior
some fatal error happend occasionaly when I test the code in multiple threads:

Describe the expected behavior
How to avoid this situation? This fatal error killed the entire Java service
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

1.code:
`// get tensorflow serving response
List servingResultList = responseFaceDet.getOutputsMap().get(concat).getFloatValList();
Float[] servingResultArray = servingResultList.toArray(new Float[15232]);
FloatDataBuffer floatDataBuffer = NioDataBufferFactory.create(FloatBuffer.allocate(15232));
floatDataBuffer.write(servingResultArray);
FloatNdArray servingResultNdArray = NdArrays.wrap(Shape.of(896, 17), floatDataBuffer);
float[][] servingBoxResultArray = StdArrays.array2dCopyOf(servingResultNdArray);

        // prepare data for non max suppression
        FloatNdArray scores1dArray = servingResultNdArray.slice(all(), at(16));
        FloatNdArray a = servingResultNdArray.slice(all(), seq(1, 0));
        FloatNdArray b = servingResultNdArray.slice(all(), seq(3, 2));

        //generate constant max filled  2
        FloatDataBuffer constantArrayBuffer = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
        Float[] fConst2 = new Float[1796];
        Arrays.fill(fConst2, 2.f);
        constantArrayBuffer.write(fConst2);
        FloatNdArray constantArray = NdArrays.wrap(Shape.of(896, 2), constantArrayBuffer);

        //generate constant max filled  0
        FloatDataBuffer constantArrayBuffer0 = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
        Float[] fConst0 = new Float[1796];
        Arrays.fill(fConst0, 0.f);
        constantArrayBuffer0.write(fConst0);
        FloatNdArray constantArray0 = NdArrays.wrap(Shape.of(896, 2), constantArrayBuffer0);

        //generate constant max filled  max value
        FloatDataBuffer constantArrayBufferNull = NioDataBufferFactory.create(FloatBuffer.allocate(1796));
        Float[] fConstNull = new Float[1796];
        Arrays.fill(fConstNull, 1230f);
        constantArrayBufferNull.write(fConstNull);
        FloatNdArray constantArrayNull = NdArrays.wrap(Shape.of(896, 2), constantArrayBufferNull);

        try (EagerSession session = EagerSession.create()) {
            Ops tf = Ops.create(session);
            Div<TFloat32> div = tf.math.div(tf.constant(b), tf.constant(constantArray));
            Neg<TFloat32> neg = tf.math.neg(div);
            Add<TFloat32> add = tf.math.add(neg, tf.constant(a));
            Add<TFloat32> add2 = tf.math.add(div, tf.constant(a));
            ClipByValue<TFloat32> clipByValue = tf.clipByValue(add, tf.constant(constantArray0), tf.constant(constantArrayNull));
            Concat<TFloat32> finalBoxes = tf.concat(Arrays.asList(clipByValue, add2), tf.constant(1));
            NonMaxSuppression nonMaxSuppression = tf.image.nonMaxSuppression(finalBoxes, tf.constant(scores1dArray), tf.constant(5), tf.constant(0.5f), tf.constant(FACE_THRESHOLD), tf.constant(0.01f));
            Output<TInt32> indiceOutput = nonMaxSuppression.selectedIndices();
            int[] indexNMS = StdArrays.array1dCopyOf(indiceOutput.data());
            Output<TFloat32> scoreOutput = nonMaxSuppression.selectedScores();
            float[] scoreNMS = StdArrays.array1dCopyOf(scoreOutput.data());
            List scoreList = new ArrayList();
            float[] boxArray;
            if (scoreNMS.length > 0) {
                boxArray = servingBoxResultArray[indexNMS[0]];
                scoreList.add(scoreNMS[0]);
                log.info("scoreList index 0 = {}", scoreNMS[0]);
            } else {
                boxArray = new float[4];
            }
            float offsetXOri = (float) dx / (float) FACEDET_SIZE_NEW;
            float offsetYOri = (float) dy / (float) FACEDET_SIZE_NEW;
            float scaleXOri = (float) FACEDET_SIZE_NEW / (float) faceDetWidth;
            float scaleYOri = (float) FACEDET_SIZE_NEW / (float) faceDetHeight;
            // get Top1 highest score Box
            centX = (int) ((boxArray[0] - offsetXOri) * scaleYOri * finalHeight);
            centY = (int) ((boxArray[1] - offsetYOri) * scaleXOri * finalWidth);
            responseWidth = (int) (boxArray[2] * scaleYOri * finalHeight);
            responseHeight = (int) (boxArray[3] * scaleXOri * finalWidth);

            CvRect cvRect2 = new CvRect();
            cvRect2.x((centX - responseWidth / 2));
            cvRect2.y((centY - responseHeight / 2));
            cvRect2.width(responseWidth);
            cvRect2.height(responseHeight);

            Position topLeft = Position.newBuilder().setX((centX - responseWidth / 2)).setY((centY - responseHeight / 2)).build();
            Position bottomRight = Position.newBuilder().setX((centX + responseWidth / 2)).setY((centY + responseHeight / 2)).build();

            facePosition.add(topLeft);
            facePosition.add(bottomRight);
        } catch (StatusRuntimeException e) {
            if (e.getStatus().getCode() == Status.Code.DEADLINE_EXCEEDED) {
                log.warn("detectMotion face model postprocessing failed!!!", e);
                return ModelCheckResponse.newBuilder()
                        .setFace(true)
                        .build();
            } else {
                throw e;
            }
        }`

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

2.hs_err_pid24993.log
A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f0d1d772691, pid=13893, tid=0x00007f0c76afa700

JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
Problematic frame:
C [libc.so.6+0x16f691] __strlen_sse2_pminub+0x11

Core dump written. Default location: /opt/app/logs/bd-pica-api/core or core.13893

If you would like to submit a bug report, please visit:
http://bugreport.java.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.
--------------- T H R E A D ---------------

Current thread (0x00007f0c88389800): JavaThread "grpc-executor-48" daemon [_thread_in_native, id=22083, stack(0x00007f0c769fa000,0x00007f0c76afb000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f0b502e32e0

Registers:
RAX=0x0000000000000000, RBX=0x00007f0c76af9198, RCX=0x0000000000000020, RDX=0x00007f0b57ff8900
RSP=0x00007f0c76af9168, RBP=0x00007f0c76af91e0, RSI=0x0000000000000000, RDI=0x00007f0b502e32e0
R8 =0x00007f0c8c63c290, R9 =0x0000000000000002, R10=0x0000000000000000, R11=0x00007f0d1d5d6f88
R12=0x00007f0c76af91f8, R13=0x00007f0b502e32e0, R14=0x00007f0c8c63c110, R15=0x00007f0bd3c49510
RIP=0x00007f0d1d772691, EFLAGS=0x0000000000010287, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f0c76af9168)
0x00007f0c76af9168: 00007f0c4ecc0153 0000000000000002
0x00007f0c76af9178: 00007f0c8c63c290 00007f0b57ff8900
0x00007f0c76af9188: 00007f0c8ffbb540 00000007c02be468
0x00007f0c76af9198: 0000000000000000 0000000000000000
0x00007f0c76af91a8: 0000000000000000 00000007bf985ad8
0x00007f0c76af91b8: 00000007bf985b00 00007f0c8c683c00
0x00007f0c76af91c8: 00000007bf985b5c 00007f0c76af9250
0x00007f0c76af91d8: 00007f0c88389800 00007f0c76af9210
0x00007f0c76af91e8: 00007f0c4ec38138 00007f0c76af9260
0x00007f0c76af91f8: 0000000000000000 00000007bf985b00
0x00007f0c76af9208: 0000000000000000 00007f0c76af9260
0x00007f0c76af9218: 00007f0d09122d4d 00000007bf985720
0x00007f0c76af9228: 00000007bf985c10 00000007bf985b68
0x00007f0c76af9238: 00000007bf985cc0 0000001a48257a79
0x00007f0c76af9248: 0000000000000003 00000006c0baf2e8
0x00007f0c76af9258: 00007f0c5593e4b7 00000007bf985cf0
0x00007f0c76af9268: 00007f0d097f3964 00000007bf985cc0
0x00007f0c76af9278: 00000007bf9789e8 00000007bf985ad8
0x00007f0c76af9288: 00000007bf985720 00000007bf985b68
0x00007f0c76af9298: 00000007bf985c10 00000007bf985cc0
0x00007f0c76af92a8: 00007f0c4ece5f7f 00000007bf9859f8
0x00007f0c76af92b8: 00000007bf985990 00000006c0baf2e8
0x00007f0c76af92c8: 00000007bf9859f4 00000000f7f30b45
0x00007f0c76af92d8: 00007f0d08ec31a0 00000007bf985a28
0x00007f0c76af92e8: 0000000000000000 00000007bf985ac0
0x00007f0c76af92f8: 00007f0d09b2d4ac 00000000f7f30b45
0x00007f0c76af9308: 00007f0d091e8278 00000007bf985968
0x00007f0c76af9318: 00000007bf985570 00000007bf985a58
0x00007f0c76af9328: 00000007bf985980 0000001a48257a79
0x00007f0c76af9338: 0000000000000003 00000006c0baf2e8
0x00007f0c76af9348: 00000007bf96dd28 00000007bf985ac0
0x00007f0c76af9358: 00007f0d0933aa6c 00000007bf985ac0

Instructions: (pc=0x00007f0d1d772691)
0x00007f0d1d772671: c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48
0x00007f0d1d772681: 31 c0 89 f9 83 e1 3f 66 0f ef c0 83 f9 30 77 1d
0x00007f0d1d772691: f3 0f 6f 0f 66 0f 74 c1 66 0f d7 d0 85 d2 0f 85
0x00007f0d1d7726a1: 4e 02 00 00 48 89 f8 48 83 e0 f0 eb 24 48 89 f8

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x00007f0c76af9198 is pointing into the stack for thread: 0x00007f0c88389800
RCX=0x0000000000000020 is an unknown value
RDX=0x00007f0b57ff8900 is an unknown value
RSP=0x00007f0c76af9168 is pointing into the stack for thread: 0x00007f0c88389800
RBP=0x00007f0c76af91e0 is pointing into the stack for thread: 0x00007f0c88389800
RSI=0x0000000000000000 is an unknown value
RDI=0x00007f0b502e32e0 is an unknown value
R8 =0x00007f0c8c63c290 is an unknown value
R9 =0x0000000000000002 is an unknown value
R10=0x0000000000000000 is an unknown value
R11=0x00007f0d1d5d6f88: <offset 0xfc3f88> in /opt/programs/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so at 0x00007f0d1c613000
R12=0x00007f0c76af91f8 is pointing into the stack for thread: 0x00007f0c88389800
R13=0x00007f0b502e32e0 is an unknown value
R14=0x00007f0c8c63c110 is an unknown value
R15=0x00007f0bd3c49510 is an unknown value

Stack: [0x00007f0c769fa000,0x00007f0c76afb000], sp=0x00007f0c76af9168, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x16f691] __strlen_sse2_pminub+0x11
C [libtensorflow_cc.so.2+0x2385138] TFE_Execute+0x28
J 3768 org.tensorflow.internal.c_api.global.tensorflow.TFE_Execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/bytedeco/javacpp/PointerPointer;Lorg/bytedeco/javacpp/IntPointer;Lorg/tensorflow/internal/c_api/TF_Status;)V (0 bytes) @ 0x00007f0d09122d4d [0x00007f0d09122c40+0x10d]
J 5826 C2 org.tensorflow.EagerOperationBuilder.execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/tensorflow/EagerSession;)[Lorg/tensorflow/internal/c_api/TFE_TensorHandle; (210 bytes) @ 0x00007f0d097f3964 [0x00007f0d097f36e0+0x284]
J 6675 C2 org.tensorflow.EagerOperationBuilder.build()Lorg/tensorflow/EagerOperation; (57 bytes) @ 0x00007f0d09b2d4ac [0x00007f0d09b2d460+0x4c]
J 8643 C2 org.tensorflow.op.MathOps.div(Lorg/tensorflow/Operand;Lorg/tensorflow/Operand;)Lorg/tensorflow/op/math/Div; (10 bytes) @ 0x00007f0d0933aa6c [0x00007f0d09339e80+0xbec]
J 7481 C2 com.em.api.algo.impl.FaceProcessorSerivceImpl.getModelInfo(Lcom/em/api/face/ModelCheckRequest;)Lcom/em/api/face/ModelCheckResponse; (2376 bytes) @ 0x00007f0d09f98068 [0x00007f0d09f94140+0x3f28]
J 8623 C2 com.em.api.face.ModelCheckServiceGrpc$MethodHandlers.invoke(Ljava/lang/Object;Lio/grpc/stub/StreamObserver;)V (48 bytes) @ 0x00007f0d0987d630 [0x00007f0d0987d000+0x630]
J 8652 C2 io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext()V (73 bytes) @ 0x00007f0d08afc0f4 [0x00007f0d08afc020+0xd4]
J 7467 C2 io.grpc.internal.ContextRunnable.run()V (35 bytes) @ 0x00007f0d09e77828 [0x00007f0d09e77620+0x208]
J 7931 C2 io.grpc.internal.SerializingExecutor.run()V (99 bytes) @ 0x00007f0d0904b930 [0x00007f0d0904b6a0+0x290]
J 9207 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f0d096d3994 [0x00007f0d096d36e0+0x2b4]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x690206] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x690711] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V [libjvm.so+0x690bb7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V [libjvm.so+0x7278f0] thread_entry(JavaThread*, Thread*)+0xa0
V [libjvm.so+0xa742b3] JavaThread::thread_main_inner()+0x103
V [libjvm.so+0xa743fc] JavaThread::run()+0x11c
V [libjvm.so+0x924ea8] java_start(Thread*)+0x108
C [libpthread.so.0+0x7ea5] start_thread+0xc5

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 3768 org.tensorflow.internal.c_api.global.tensorflow.TFE_Execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/bytedeco/javacpp/PointerPointer;Lorg/bytedeco/javacpp/IntPointer;Lorg/tensorflow/internal/c_api/TF_Status;)V (0 bytes) @ 0x00007f0d09122ccb [0x00007f0d09122c40+0x8b]
J 5826 C2 org.tensorflow.EagerOperationBuilder.execute(Lorg/tensorflow/internal/c_api/TFE_Op;Lorg/tensorflow/EagerSession;)[Lorg/tensorflow/internal/c_api/TFE_TensorHandle; (210 bytes) @ 0x00007f0d097f3964 [0x00007f0d097f36e0+0x284]
J 6675 C2 org.tensorflow.EagerOperationBuilder.build()Lorg/tensorflow/EagerOperation; (57 bytes) @ 0x00007f0d09b2d4ac [0x00007f0d09b2d460+0x4c]
J 8643 C2 org.tensorflow.op.MathOps.div(Lorg/tensorflow/Operand;Lorg/tensorflow/Operand;)Lorg/tensorflow/op/math/Div; (10 bytes) @ 0x00007f0d0933aa6c [0x00007f0d09339e80+0xbec]
J 7481 C2 com.em.api.algo.impl.FaceProcessorSerivceImpl.getModelInfo(Lcom/em/api/face/ModelCheckRequest;)Lcom/em/api/face/ModelCheckResponse; (2376 bytes) @ 0x00007f0d09f98068 [0x00007f0d09f94140+0x3f28]
J 8623 C2 com.em.api.face.ModelCheckServiceGrpc$MethodHandlers.invoke(Ljava/lang/Object;Lio/grpc/stub/StreamObserver;)V (48 bytes) @ 0x00007f0d0987d630 [0x00007f0d0987d000+0x630]
J 8652 C2 io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext()V (73 bytes) @ 0x00007f0d08afc0f4 [0x00007f0d08afc020+0xd4]
J 7467 C2 io.grpc.internal.ContextRunnable.run()V (35 bytes) @ 0x00007f0d09e77828 [0x00007f0d09e77620+0x208]
J 7931 C2 io.grpc.internal.SerializingExecutor.run()V (99 bytes) @ 0x00007f0d0904b930 [0x00007f0d0904b6a0+0x290]
J 9207 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f0d096d3994 [0x00007f0d096d36e0+0x2b4]
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从提供的多线程 Java 代码和 hs_err_pid24993.log 开始,重点关注原生 TFE_Execute 帧和间歇性 SIGSEGV。为 Ubuntu 18.04 上的 TF Java 0.2.0 建立一个最小可复现测试;完成的标准是解释清楚原生崩溃,并确定一个经过验证的规避方法或修复方案。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
backend
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。