microsoft / microsoft/SynapseML
LightGBMClassifier: multiclass training, workers out of sync and early termination
@imatiach-msft is already working on this.
Since May 16, 2019.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
I am training a lightGBM classifier on a dataset with 18 classes, ~ 2M rows and ~5900 columns. The data is pretty sparse (density ~ 15%). The class distribution is unbalanced.
```
lgbm = LightGBMClassifier(objective='multiclass', parallelism='voting_parallel')
model = lgbm.fit(train)
```
I have a 20 node yarn cluster, each node has 8 cores and 64G RAM. Memory is plenty for this dataset.
Looking at the executor log, I see that one of the lightGBm worker goes out of sync with the rest. If every worker is on iteration x, this worker is at a higher iteration (x + y). Eventually, I get this error,
```
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:14 INFO LightGBMClassifier: LightGBM running iteration: 99 with result: 0 and is finished: false
[LightGBM] [Fatal] Socket recv error, code: 104
19/05/16 20:12:15 WARN LightGBMClassifier: LightGBM reached early termination on one worker, stopping training on worker. This message should rarely occur
```
Notice all workers are on iteration 82 except one which is on 99. The program continues to run, but not sure how many boosting iterations were run. Also, the warning printed towards the end points something not working right.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.