when I use angel.task.data.storage.level=disk, worker failed by java.lang.NullPointerException at at com.tencent.angel.worker.storage.DiskDataBlock.resetReadIndex
- Langage dominant
- Java
- Étoiles
- 6.8k
- Forks
- 1.6k
- Merge moyen
- 44 min
- PR mergées (30 j)
- 1
Description
when I use angel.task.data.storage.level=disk, worker failed by java.lang.NullPointerException at at com.tencent.angel.worker.storage.DiskDataBlock.resetReadIndex
my job submit script is
${ANGEL_HOME}/bin/angel-submit \
--angel.app.submit.class com.tencent.angel.ml.core.graphsubmit.GraphRunner \
--ml.model.class.name com.tencent.angel.ml.classification.FactorizationMachines \
--ml.feature.index.range 700000000 \
--ml.data.type libsvm \
--ml.learn.rate 0.1 \
--ml.fm.rank 4 \
--ml.epoch.num 50 \
--ml.num.update.per.epoch 50 \
--action.type train \
--angel.am.memory.gb 12 \
--angel.am.cpu.vcores 5 \
--angel.workergroup.number 1 \
--angel.worker.memory.gb 15 \
--angel.worker.task.number 1 \
--angel.worker.log.level INFO \
--ml.model.type T_DOUBLE_SPARSE \
--angel.ps.number 30 \
--angel.ps.memory.gb 20 \
--angel.ps.cpu.vcores 5 \
--angel.am.java.opts "-Xms8g -Xmx8g" \
--angel.worker.java.opts "-Xms16g -Xmx16g" \
--angel.ps.java.opts "-Xms16g -Xmx16g" \
--angel.job.name xxx \
--queue root.${queue} \
--angel.train.data.path ${BUSINESS_USER_ATTENTION_ROOT_PATH}/matching/dnn/ins/log_date=${ds}/part-00000.gz \
--angel.log.path ${BUSINESS_USER_ATTENTION_ROOT_PATH}/matching/dnn/train/log/log_date=${ds} \
--angel.save.model.path /tmp/angel/business_user_attention_user_interest_model/model/ \
--angel.task.data.storage.level "disk" \
--angel.output.path.deleteonexist true
the error is
killed and failed workergroup is over tolerate 0.0There are some Workers failed
failed workergroups:
WorkerGroup_0. Worker_0_0 failed. WorkerAttempt_0_0_0 failed due to: taskid=task_0, state=FAILED, diagnostics=[task runner error: java.lang.NullPointerException
at com.tencent.angel.worker.storage.DiskDataBlock.resetReadIndex(DiskDataBlock.java:235)
at com.tencent.angel.ml.core.graphsubmit.GraphLearner.getBathDataIterator(GraphLearner.scala:194)
at com.tencent.angel.ml.core.graphsubmit.GraphLearner.train(GraphLearner.scala:158)
at com.tencent.angel.ml.core.graphsubmit.GraphLearner.train(GraphLearner.scala:109)
at com.tencent.angel.ml.core.graphsubmit.GraphTrainTask.train(GraphTrainTask.scala:48)
at com.tencent.angel.ml.core.TrainTask.run(TrainTask.scala:45)
at com.tencent.angel.worker.task.Task.runUser(Task.java:92)
at com.tencent.angel.worker.task.Task.run(Task.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.