microsoft / microsoft/SynapseML

LightGBM and dynamic allocation

Open
#319 7 comments 2 reactions 1 assignee Claimed by @imatiach-msft View on GitHub
area/lightgbm enhancement high priority
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

Hello,

We are currently trying to use MMLSpark and LightGBM in our Spark cluster. We usually use default dynamic allocation of executors when starting jobs, but LightGBM seems to not work with this setting (the main Thread used to wait for executors connection only replies to the first ones created during the initialisation of the main Spark context).

How is it possible to make LightGBM work in a dynamic executors allocation way?

Thanks!

EDIT: we are using the current `master` branch

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.