microsoft / microsoft/SynapseML

LightGBMClassifier: multiclass training, workers out of sync and early termination

Open
#569 4 comments 0 reactions 1 assignee View on GitHub

@imatiach-msft is already working on this.

Since May 16, 2019.

area/lightgbm bug high priority
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

I am training a lightGBM classifier on a dataset with 18 classes, ~ 2M rows and ~5900 columns. The data is pretty sparse (density ~ 15%). The class distribution is unbalanced.
```
lgbm = LightGBMClassifier(objective='multiclass', parallelism='voting_parallel')
model = lgbm.fit(train)
```
I have a 20 node yarn cluster, each node has 8 cores and 64G RAM. Memory is plenty for this dataset.

Looking at the executor log, I see that one of the lightGBm worker goes out of sync with the rest. If every worker is on iteration x, this worker is at a higher iteration (x + y). Eventually, I get this error,
```
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM running iteration: 82 with result: 0 and is finished: false
19/05/16 20:12:04 INFO LightGBMClassifier: LightGBM worker calling LGBM_BoosterUpdateOneIter
19/05/16 20:12:14 INFO LightGBMClassifier: LightGBM running iteration: 99 with result: 0 and is finished: false
[LightGBM] [Fatal] Socket recv error, code: 104
19/05/16 20:12:15 WARN LightGBMClassifier: LightGBM reached early termination on one worker, stopping training on worker. This message should rarely occur
```

Notice all workers are on iteration 82 except one which is on 99. The program continues to run, but not sure how many boosting iterations were run. Also, the warning printed towards the end points something not working right.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.