microsoft / microsoft/SynapseML
mml spark Incorrect lightgbm predictions
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Pretext**
I trained the model using the dataset created from a **pandas df** viz
` train_dataset = lgb.Dataset(
df,
label=df["label"],
weight=df["weight"].values,
free_raw_data=True,
)`
and then training
` model = lgb.train(
config["hyperparams"],
train_dataset,
verbose_eval=0,
valid_sets=[test_dataset],
)`
I write the model string to a file on hdfs, named model.lgb
**Objective**
To use this file on hdfs to load with mmlSpark and make predictions.
**Expectation**
The predictions on the pandas df should match with the predictions on spark df in mmlSpark.
What's actually happening is predictions don't match?
The code that I am running on pyspark is:
```
from mmlspark.lightgbm import LightGBMClassifier, LightGBMClassificationModel
from pyspark.ml.feature import StringIndexer, VectorAssembler
model_path = "hdfs://nameservice1/user/admin/model.lgb"
model = LightGBMClassificationModel.loadNativeModelFromFile(model_path)
df = spark.sql("select * from rpm_misc.mml_test")
ft ==> list of features
features=[]
for f in ft:
string_indexer = StringIndexer(inputCol=f, outputCol=f + "_index")
model_si = string_indexer.fit(data)
data = model_si.transform(data)
features.append(f + "_index")
vector_assembler = VectorAssembler(inputCols=eatures, outputCol="features")
data = vector_assembler.transform(data)
preds = model.transform(data)
pred_probs = preds.select("probability").rdd.flatMap(lambda x: x).collect()
print(pred_probs)
```
What am I doing wrong?
--repositories https://mmlspark.azureedge.net/maven --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3
Contributor guide
Assessment
This issue has not been assessed yet.