Angel-ML / Angel-ML/angel

when I use angel.task.data.storage.level=disk, worker failed by java.lang.NullPointerException at at com.tencent.angel.worker.storage.DiskDataBlock.resetReadIndex

Ouverte
#865 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Java
Étoiles
6.8k
Forks
1.6k
Merge moyen
44 min
PR mergées (30 j)
1

Description

when I use angel.task.data.storage.level=disk, worker failed by java.lang.NullPointerException at at com.tencent.angel.worker.storage.DiskDataBlock.resetReadIndex
my job submit script is
${ANGEL_HOME}/bin/angel-submit \
--angel.app.submit.class com.tencent.angel.ml.core.graphsubmit.GraphRunner \
--ml.model.class.name com.tencent.angel.ml.classification.FactorizationMachines \
--ml.feature.index.range 700000000 \
--ml.data.type libsvm \
--ml.learn.rate 0.1 \
--ml.fm.rank 4 \
--ml.epoch.num 50 \
--ml.num.update.per.epoch 50 \
--action.type train \
--angel.am.memory.gb 12 \
--angel.am.cpu.vcores 5 \
--angel.workergroup.number 1 \
--angel.worker.memory.gb 15 \
--angel.worker.task.number 1 \
--angel.worker.log.level INFO \
--ml.model.type T_DOUBLE_SPARSE \
--angel.ps.number 30 \
--angel.ps.memory.gb 20 \
--angel.ps.cpu.vcores 5 \
--angel.am.java.opts "-Xms8g -Xmx8g" \
--angel.worker.java.opts "-Xms16g -Xmx16g" \
--angel.ps.java.opts "-Xms16g -Xmx16g" \
--angel.job.name xxx \
--queue root.${queue} \
--angel.train.data.path ${BUSINESS_USER_ATTENTION_ROOT_PATH}/matching/dnn/ins/log_date=${ds}/part-00000.gz \
--angel.log.path ${BUSINESS_USER_ATTENTION_ROOT_PATH}/matching/dnn/train/log/log_date=${ds} \
--angel.save.model.path /tmp/angel/business_user_attention_user_interest_model/model/ \
--angel.task.data.storage.level "disk" \
--angel.output.path.deleteonexist true

the error is
killed and failed workergroup is over tolerate 0.0There are some Workers failed
failed workergroups:
WorkerGroup_0. Worker_0_0 failed. WorkerAttempt_0_0_0 failed due to: taskid=task_0, state=FAILED, diagnostics=[task runner error: java.lang.NullPointerException
at com.tencent.angel.worker.storage.DiskDataBlock.resetReadIndex(DiskDataBlock.java:235)
at com.tencent.angel.ml.core.graphsubmit.GraphLearner.getBathDataIterator(GraphLearner.scala:194)
at com.tencent.angel.ml.core.graphsubmit.GraphLearner.train(GraphLearner.scala:158)
at com.tencent.angel.ml.core.graphsubmit.GraphLearner.train(GraphLearner.scala:109)
at com.tencent.angel.ml.core.graphsubmit.GraphTrainTask.train(GraphTrainTask.scala:48)
at com.tencent.angel.ml.core.TrainTask.run(TrainTask.scala:45)
at com.tencent.angel.worker.task.Task.runUser(Task.java:92)
at com.tencent.angel.worker.task.Task.run(Task.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.