Angel-ML / Angel-ML/angel

A Bug In LossMetric which will lead to train.loss less then zero

Aperta
#870 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
6.8k
Fork
1.6k
Merge medio
44m
PR unite (30g)
1

Descrizione

In LossMetric class, the sampleNum is Int and when sampleNum larger than 2^32, the sampleNum will overflow. the code is

class LossMetric(var sampleNum: Int) extends Metric {
var loss: Double = 0.0

override def merge(other: this.type): this.type = {
this.sampleNum += other.sampleNum
this.loss += other.loss
this
}
the phenomenon is
19/09/17 15:12:53 INFO client.AngelClient: Epoch: 1. Metrics={"train.loss":-17.665040,"validate.loss":0.313661}
19/09/17 15:31:06 INFO client.AngelClient: Epoch: 2. Metrics={"train.loss":-16.968140,"validate.loss":0.300109}
19/09/17 15:42:20 INFO client.AngelClient: Epoch: 3. Metrics={"train.loss":-16.734170,"validate.loss":0.293660}
19/09/17 15:51:09 INFO client.AngelClient: Epoch: 4. Metrics={"train.loss":-16.385350,"validate.loss":0.284851}
19/09/17 16:01:57 INFO client.AngelClient: Epoch: 5. Metrics={"train.loss":-16.013730,"validate.loss":0.278738}
19/09/17 16:26:41 INFO client.AngelClient: Epoch: 6. Metrics={"train.loss":-15.925190,"validate.loss":0.278801}
19/09/17 17:06:11 INFO client.AngelClient: Epoch: 7. Metrics={"train.loss":-15.980800,"validate.loss":0.280390}
19/09/17 17:18:53 INFO client.AngelClient: Epoch: 8. Metrics={"train.loss":-15.683660,"validate.loss":0.276296}
19/09/17 17:30:14 INFO client.AngelClient: Epoch: 9. Metrics={"train.loss":-15.226160,"validate.loss":0.269379}
19/09/17 17:42:57 INFO client.AngelClient: Epoch: 10. Metrics={"train.loss":-14.883470,"validate.loss":0.264469}

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.