java.lang.OutOfMeomoryError: GC overhead limit exceeded with DeepFM
- Lingua principale
- Java
- Stelle
- 6.8k
- Fork
- 1.6k
- Merge medio
- 44m
- PR unite (30g)
- 1
Descrizione
I wanted to trian a deepfm model with about 100GB samples. After I ran the angel-submmit, Yarn.Client started pass sucessfully. But train process has been always maintaining the 0-th epoch. And then the job failed, thorwing the error:
workergoup_17_17 failed due to heartbeat timeout, taskid_task_17, state=FAILED, diagnostics=[task runner error: java.lang.OutOfMemotyError: GC overhead limit exceeded]
The numbers of workergroup and PS are 32 and 4, with 32GB and 64GB memory respectively. And
--angel.am.java.opts "-Xms8g -Xmx8g" \
--angel.worker.java.opts "-Xms16g -Xmx16g" \
--angel.ps.java.opts "-Xms16g -Xmx16g"
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.