关于GBDT模型训练value值出现NaN之后的卡住问题

这边GBDT模型训练有一个新的问题，我在训练多棵树的时候，会出现训练中间卡住的现象。但是程序并不会down掉，也没有报错。我在观察日志的时候发现卡住日志的部分有节点value值为NaN的情况，我在想是不是因为这个原因。还是其他原因？

现训练日志如下：2019-09-24 12:02:57,745 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: ------Find split------
2019-09-24 12:02:57,745 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: Task[0] responsible tree node: [3]
2019-09-24 12:02:57,745 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 460
2019-09-24 12:02:58,744 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 460 over
2019-09-24 12:02:58,753 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: Pull histogram from PS cost 1008 ms
2019-09-24 12:02:58,753 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: Best split of node[3]: feature[45], value[NaN], losschg[2212614.000000]
2019-09-24 12:02:58,758 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: Find split cost: 1013 ms
2019-09-24 12:02:58,908 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: clock and flush matrices [gbdt.split.value, gbdt.node.grad.stats, gbdt.split.feature, gbdt.split.gain] cost 150 ms
2019-09-24 12:02:58,909 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.GBDTLearner: ******Current phase: AFTER_SPLIT, clock[504]******
2019-09-24 12:02:58,909 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: ------After split------
2019-09-24 12:02:58,909 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461
2019-09-24 12:02:58,949 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461 over
2019-09-24 12:02:58,963 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461
2019-09-24 12:02:58,963 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461 over
2019-09-24 12:02:58,973 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461
2019-09-24 12:02:58,973 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461 over
2019-09-24 12:02:58,994 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461
2019-09-24 12:02:58,994 INFO [pool-5-thread-1] com.tencent.angel.psagent.consistency.ConsistencyController: wait for clock 461 over
2019-09-24 12:02:59,015 INFO [pool-5-thread-1] com.tencent.angel.ml.GBDT.algo.GBDTController: Get split result from PS cost 106 ms
2019-09-24 12:02:59,016 INFO [pool-9-thread-6] com.tencent.angel.ml.GBDT.algo.AfterSplitThread: Active node[3]: split feature[45] value[NaN], lossChg[2212614.000000], sumGrad[29335740.000000], sumHess[27589250.000000]
2019-09-24 12:02:59,030 INFO [pool-9-thread-11] com.tencent.angel.ml.GBDT.algo.AfterSplitThread: Active node[5]: split feature[42] value[-2.000000], lossChg[1992325.625000], sumGrad[-13305986.000000], sumHess[42665720.000000]
2019-09-24 12:02:59,030 INFO [pool-9-thread-16] com.tencent.angel.ml.GBDT.algo.AfterSplitThread: Active node[6]: split feature[42] value[-2.000000], lossChg[1765322.000000], sumGrad[-33581768.000000], sumHess[42314952.000000]
2019-09-24 12:02:59,043 INFO [pool-9-thread-5] com.tencent.angel.ml.GBDT.algo.AfterSplitThread: Active node[4]: split feature[3] value[0.012293], lossChg[2749910.500000], sumGrad[6899572.000000], sumHess[48063624.000000]
2019-09-25 00:12:28,029 WARN [Worker Heartbeat] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException: No such process
2019-09-25 05:31:39,058 WARN [Worker Heartbeat] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException: No such process
2019-09-25 12:43:55,286 WARN [Worker Heartbeat] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException: No such process
2019-09-25 13:45:49,951 WARN [Worker Heartbeat] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException: No such process