microsoft / microsoft/SynapseML

mml spark Incorrect lightgbm predictions

Open
#1,513 2 comments 0 reactions 1 assignee Claimed by @imatiach-msft View on GitHub
area/lightgbm bug
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

**Pretext**
I trained the model using the dataset created from a **pandas df** viz
` train_dataset = lgb.Dataset(
df,
label=df["label"],
weight=df["weight"].values,
free_raw_data=True,
)`

and then training
` model = lgb.train(
config["hyperparams"],
train_dataset,
verbose_eval=0,
valid_sets=[test_dataset],
)`

I write the model string to a file on hdfs, named model.lgb

**Objective**
To use this file on hdfs to load with mmlSpark and make predictions.

**Expectation**

The predictions on the pandas df should match with the predictions on spark df in mmlSpark.

What's actually happening is predictions don't match?

The code that I am running on pyspark is:

```
from mmlspark.lightgbm import LightGBMClassifier, LightGBMClassificationModel
from pyspark.ml.feature import StringIndexer, VectorAssembler
model_path = "hdfs://nameservice1/user/admin/model.lgb"
model = LightGBMClassificationModel.loadNativeModelFromFile(model_path)

df = spark.sql("select * from rpm_misc.mml_test")

ft ==> list of features
features=[]

for f in ft:
string_indexer = StringIndexer(inputCol=f, outputCol=f + "_index")
model_si = string_indexer.fit(data)
data = model_si.transform(data)
features.append(f + "_index")

vector_assembler = VectorAssembler(inputCols=eatures, outputCol="features")
data = vector_assembler.transform(data)
preds = model.transform(data)

pred_probs = preds.select("probability").rdd.flatMap(lambda x: x).collect()
print(pred_probs)
```

What am I doing wrong?

--repositories https://mmlspark.azureedge.net/maven --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.