microsoft / microsoft/SynapseML
[BUG] LightGBM is failing on a cluster with 2 executors or more
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
### SynapseML version
1.1.3
### System information
- **Language version** (e.g. python 3.8, scala 2.12): Python 3.12.3, Scala 2.13 and Scala 2.12
- **Spark Version** (e.g. 3.2.3): 4.1.0
- **Spark Platform** (e.g. Synapse, Databricks): Databricks
### Describe the problem
When running this code on a single node or a cluster with one executors this works fine regardless of the cardinality of the categorical column. However, as soon as the number of executors are 2 or more and cardinality of the categorical column is more than 56 this will fail.
We have compiled the SynapseML LGBM for scala 2.13 using `com.microsoft.ml.lightgbm:lightgbmlib:3.3.500` instead of `com.microsoft.ml.lightgbm:lightgbmlib:3.3.510` and this resolved the issue.
[SynapseML213-success-Cluster-single-executor-high-cardinality.ipynb](https://github.com/user-attachments/files/31280790/SynapseML213-success-Cluster-single-executor-high-cardinality.ipynb)
[SynapseML213-Success-Cluster-2+executors-56-cardinality.ipynb](https://github.com/user-attachments/files/31280791/SynapseML213-Success-Cluster-2%2Bexecutors-56-cardinality.ipynb)
[SynapseML213-Failure-Cluster-2+executors-high-cardinality.ipynb](https://github.com/user-attachments/files/31280792/SynapseML213-Failure-Cluster-2%2Bexecutors-high-cardinality.ipynb)
### Code to reproduce issue
```
from synapse.ml.lightgbm import LightGBMRegressor
from pyspark.ml.feature import StringIndexer, VectorAssembler
# Generate synthetic data with 100,000 rows and 6 categorical columns
import numpy as np
import pandas as pd
num_rows = 100000
cat7_values = [i for i in range(200)] # 200 distinct values for cat7
df_pd = pd.DataFrame({
"cat1": np.random.choice(['A', 'B', 'C'], num_rows),
"cat2": np.random.choice(['X', 'Y', 'Z'], num_rows),
"cat3": np.random.choice(['foo', 'bar', 'baz'], num_rows),
"cat4": np.random.choice(['red', 'green', 'blue'], num_rows),
"cat5": np.random.choice(['dog', 'cat', 'mouse'], num_rows),
"cat6": np.random.choice(['apple', 'banana', 'cherry'], num_rows),
"cat7": np.random.choice(cat7_values, num_rows), # new high-cardinality column
"num1": np.random.randn(num_rows),
"num2": np.random.randn(num_rows),
"num3": np.random.randn(num_rows),
"num4": np.random.randn(num_rows),
"label": np.random.randn(num_rows)
})
df = spark.createDataFrame(df_pd)
# Index categorical columns
indexers = [
StringIndexer(inputCol=col, outputCol=f"{col}_idx")
for col in ["cat1", "cat2", "cat3", "cat4", "cat5", "cat6", "cat7"]
]
# Assemble features
assembler = VectorAssembler(
inputCols=["cat1_idx", "cat2_idx", "cat3_idx", "cat4_idx", "cat5_idx", "cat6_idx", "cat7_idx", "num1", "num2", "num3", "num4"],
outputCol="features"
)
# LightGBMRegressor
lgbm = LightGBMRegressor(
featuresCol="features",
labelCol="label",
categoricalSlotIndexes=[0, 1, 2, 3, 4, 5, 6],
useBarrierExecutionMode=True,
# numTasks=2
)
from pyspark.ml import Pipeline
pipeline = Pipeline(stages=indexers + [assembler, lgbm])
# df = df.repartition(2)
model = pipeline.fit(df)
```
### Other info / logs
_No response_
### What component(s) does this bug affect?
- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [x] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue
### What language(s) does this bug affect?
- [ ] `language/scala`: Scala source code
- [x] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages
### What integration(s) does this bug affect?
- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations
Contributor guide
Research direction
Start at the LightGBMRegressor pipeline entry point and run the provided success and failure notebooks against the stated Spark, Scala, Python, and Databricks versions. Compare behavior with two or more executors and categorical cardinality above 56, including the reported lightgbmlib 3.3.500 versus 3.3.510 difference. Done means the reproduced multi-executor case no longer fails without regressing the working single-executor case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scala
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100