microsoft / microsoft/SynapseML

LightGBMRegressor not deterministic with deterministic=True, seed=777, force_col_wise=True

Open
#1,886 7 comments 0 reactions 1 assignee Claimed by @imatiach-msft View on GitHub
bug question
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

### SynapseML version

com.microsoft.azure:synapseml_2.12:0.11.0

### System information

- **Language version** (e.g. python 3.8, scala 2.12): Python 3.9.5, Scala 2.12
- **Spark Version** (e.g. 3.2.3): 3.3.0
- **Spark Platform** (e.g. Synapse, Databricks): Databricks

Databricks version is "11.3 LTS ML (includes Apache Spark 3.3.0, Scala 2.12)"

I am running up to 10 worker nodes of type "i3.4xlarge" (122 GB Memory, 16 cores) and my driver type is "i3.8xlarge" (244 GB Memory, 32 cores).

### Describe the problem

I am not sure if this is a bug or just an input I am missing. I am training on a large dataset and running the same code twice does not give me the same output. I have recreated the problem below with a toy dataset.

On smallish datasets (say 300K), I get deterministic results, but when I increase the size of the dataset (to say 3 million), results are no longer deterministic. Note that they are not **always** different so the below code will sometimes produce the same result for me. But sometimes I have prediction_one not equal to prediction_two. I would say they are unequal more often than not.

I was under the impression that to have deterministic results you needed to set:

- deterministic=True
- seed=777
- force_col_wise=True

Is there more you need to do to get deterministic results?

### Code to reproduce issue

```python
from synapse.ml.lightgbm import LightGBMRegressor
import numpy as np
import pandas as pd
from pyspark.ml.feature import VectorAssembler

# create data
np.random.seed(42)
num_features = 88
data_length = 3_000_000
data = pd.DataFrame({f'feature_{i}': np.random.random(data_length) for i in range(num_features)})
data['label'] = 0
for feature_num in range(num_features):
data[f'feature_{feature_num}'] *= np.where(np.random.random(data_length) < 0.5, 0, 1)
data['label'] += data[f'feature_{feature_num}'] ** feature_num
feature_names = [i for i in data if i != 'label']

# convert to trainable format
train = spark.createDataFrame(data)
featurizer = VectorAssembler(inputCols=feature_names, outputCol="features")
train_data = featurizer.transform(train)['label', "features"]

# fit a model on the data and then calculate its predicted value for the data
# if deterministic we would always expect this to give the same output for the same input
def fit_model_and_get_predictions(input_train_data):
model = LightGBMRegressor(
objective="regression", learningRate=0.1, numLeaves=30, deterministic=True, numIterations=200, seed=777,
).fit(input_train_data)
model.passThroughArgs = "force_col_wise=True"
predictions = [i[0] for i in model.transform(train_data).select('prediction').collect()]

return predictions

# run the same code twice
prediction_one = fit_model_and_get_predictions(train_data)
prediction_two = fit_model_and_get_predictions(train_data)

# predictions are different, below should evaluate as False
# it may sometimes evaluate as True, as the non-determinism itself seems non-deterministic,
# but a rerun (or a few reruns) should return False. For me it is False about 3 out of 4 times
prediction_one == prediction_two
```

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/cognitive`: Cognitive project
- [ ] `area/core`: Core project
- [ ] `area/deep-learning`: DeepLearning project
- [X] `area/lightgbm`: Lightgbm project
- [ ] `area/opencv`: Opencv project
- [ ] `area/vw`: VW project
- [ ] `area/website`: Website
- [ ] `area/build`: Project build system
- [ ] `area/notebooks`: Samples under notebooks folder
- [ ] `area/docker`: Docker usage
- [ ] `area/models`: models related issue

### What language(s) does this bug affect?

- [ ] `language/scala`: Scala source code
- [X] `language/python`: Pyspark APIs
- [ ] `language/r`: R APIs
- [ ] `language/csharp`: .NET APIs
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/synapse`: Azure Synapse integrations
- [ ] `integrations/azureml`: Azure ML integrations
- [X] `integrations/databricks`: Databricks integrations

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.