microsoft / microsoft/SynapseML

[BUG] LightGBM is failing on a cluster with 2 executors or more

Open
#2,674 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

### SynapseML version

1.1.3

### System information

- **Language version** (e.g. python 3.8, scala 2.12): Python 3.12.3, Scala 2.13 and Scala 2.12
- **Spark Version** (e.g. 3.2.3): 4.1.0
- **Spark Platform** (e.g. Synapse, Databricks): Databricks

### Describe the problem

When running this code on a single node or a cluster with one executors this works fine regardless of the cardinality of the categorical column. However, as soon as the number of executors are 2 or more and cardinality of the categorical column is more than 56 this will fail.

We have compiled the SynapseML LGBM for scala 2.13 using `com.microsoft.ml.lightgbm:lightgbmlib:3.3.500` instead of `com.microsoft.ml.lightgbm:lightgbmlib:3.3.510` and this resolved the issue.

[SynapseML213-success-Cluster-single-executor-high-cardinality.ipynb](https://github.com/user-attachments/files/31280790/SynapseML213-success-Cluster-single-executor-high-cardinality.ipynb)
[SynapseML213-Success-Cluster-2+executors-56-cardinality.ipynb](https://github.com/user-attachments/files/31280791/SynapseML213-Success-Cluster-2%2Bexecutors-56-cardinality.ipynb)
[SynapseML213-Failure-Cluster-2+executors-high-cardinality.ipynb](https://github.com/user-attachments/files/31280792/SynapseML213-Failure-Cluster-2%2Bexecutors-high-cardinality.ipynb)

### Code to reproduce issue

```
from synapse.ml.lightgbm import LightGBMRegressor
from pyspark.ml.feature import StringIndexer, VectorAssembler

# Generate synthetic data with 100,000 rows and 6 categorical columns
import numpy as np
import pandas as pd

num_rows = 100000
cat7_values = [i for i in range(200)] # 200 distinct values for cat7

df_pd = pd.DataFrame({
"cat1": np.random.choice(['A', 'B', 'C'], num_rows),
"cat2": np.random.choice(['X', 'Y', 'Z'], num_rows),
"cat3": np.random.choice(['foo', 'bar', 'baz'], num_rows),
"cat4": np.random.choice(['red', 'green', 'blue'], num_rows),
"cat5": np.random.choice(['dog', 'cat', 'mouse'], num_rows),
"cat6": np.random.choice(['apple', 'banana', 'cherry'], num_rows),
"cat7": np.random.choice(cat7_values, num_rows), # new high-cardinality column
"num1": np.random.randn(num_rows),
"num2": np.random.randn(num_rows),
"num3": np.random.randn(num_rows),
"num4": np.random.randn(num_rows),
"label": np.random.randn(num_rows)
})

df = spark.createDataFrame(df_pd)

# Index categorical columns
indexers = [
StringIndexer(inputCol=col, outputCol=f"{col}_idx")
for col in ["cat1", "cat2", "cat3", "cat4", "cat5", "cat6", "cat7"]
]

# Assemble features
assembler = VectorAssembler(
inputCols=["cat1_idx", "cat2_idx", "cat3_idx", "cat4_idx", "cat5_idx", "cat6_idx", "cat7_idx", "num1", "num2", "num3", "num4"],
outputCol="features"
)

# LightGBMRegressor
lgbm = LightGBMRegressor(
featuresCol="features",
labelCol="label",
categoricalSlotIndexes=[0, 1, 2, 3, 4, 5, 6],
useBarrierExecutionMode=True,
# numTasks=2
)

from pyspark.ml import Pipeline

pipeline = Pipeline(stages=indexers + [assembler, lgbm])
# df = df.repartition(2)
model = pipeline.fit(df)
```

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [x] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [x] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [ ] `integrations/databricks`: Databricks integrations

Contributor guide

Open the contributing guide

Research direction

Start at the LightGBMRegressor pipeline entry point and run the provided success and failure notebooks against the stated Spark, Scala, Python, and Databricks versions. Compare behavior with two or more executors and categorical cardinality above 56, including the reported lightgbmlib 3.3.500 versus 3.3.510 difference. Done means the reproduced multi-executor case no longer fails without regressing the working single-executor case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.