Angel-ML / Angel-ML/angel

java.lang.OutOfMeomoryError: GC overhead limit exceeded with DeepFM

Aperta
#764 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
6.8k
Fork
1.6k
Merge medio
44m
PR unite (30g)
1

Descrizione

I wanted to trian a deepfm model with about 100GB samples. After I ran the angel-submmit, Yarn.Client started pass sucessfully. But train process has been always maintaining the 0-th epoch. And then the job failed, thorwing the error:
workergoup_17_17 failed due to heartbeat timeout, taskid_task_17, state=FAILED, diagnostics=[task runner error: java.lang.OutOfMemotyError: GC overhead limit exceeded]

The numbers of workergroup and PS are 32 and 4, with 32GB and 64GB memory respectively. And
--angel.am.java.opts "-Xms8g -Xmx8g" \
--angel.worker.java.opts "-Xms16g -Xmx16g" \
--angel.ps.java.opts "-Xms16g -Xmx16g"

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.