java.lang.OutOfMeomoryError: GC overhead limit exceeded with DeepFM
未關閉
- 主要語言
- Java
- 星號
- 6.8k
- 分支
- 1.6k
- 平均合併
- 44 分鐘
- 30 天內合併 PR
- 1
描述
I wanted to trian a deepfm model with about 100GB samples. After I ran the angel-submmit, Yarn.Client started pass sucessfully. But train process has been always maintaining the 0-th epoch. And then the job failed, thorwing the error:
workergoup_17_17 failed due to heartbeat timeout, taskid_task_17, state=FAILED, diagnostics=[task runner error: java.lang.OutOfMemotyError: GC overhead limit exceeded]
The numbers of workergroup and PS are 32 and 4, with 32GB and 64GB memory respectively. And
--angel.am.java.opts "-Xms8g -Xmx8g" \
--angel.worker.java.opts "-Xms16g -Xmx16g" \
--angel.ps.java.opts "-Xms16g -Xmx16g"
貢獻指南
評估
這個 Issue 還沒有評估資料。