Angel-ML / Angel-ML/angel

java.lang.OutOfMeomoryError: GC overhead limit exceeded with DeepFM

未關閉
#764 4 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Java
星號
6.8k
分支
1.6k
平均合併
44 分鐘
30 天內合併 PR
1

描述

I wanted to trian a deepfm model with about 100GB samples. After I ran the angel-submmit, Yarn.Client started pass sucessfully. But train process has been always maintaining the 0-th epoch. And then the job failed, thorwing the error:
workergoup_17_17 failed due to heartbeat timeout, taskid_task_17, state=FAILED, diagnostics=[task runner error: java.lang.OutOfMemotyError: GC overhead limit exceeded]

The numbers of workergroup and PS are 32 and 4, with 32GB and 64GB memory respectively. And
--angel.am.java.opts "-Xms8g -Xmx8g" \
--angel.worker.java.opts "-Xms16g -Xmx16g" \
--angel.ps.java.opts "-Xms16g -Xmx16g"

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。