[jvm-packages] Issue in saving Xgboost model in spark scala and then load to the single Python model
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Hi.
I have trained XGBoost Classification model using scala ( spark 2.3 )
and then saved the model with nativeBooster.saveModel
( ref : https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html )
Then I loaded the model in Python3 and predict the probability
`spark env : spark 2.3 , xgb 0.82 `
`python env : python 3.6 , xgb 0.82 `
below is my shotly code
**Train ( Spark )**
`
val PIPELINE = new Pipeline().setStages(Array(VectorAssember_Input,XGBoostClassifier))
MDL = PIPELINE_666715c8_1acc_490f_bc46_e29f92070d23.fit(InputDataFrame)`
`MDL.write.overwrite().save(output_path)`
`val model = MDL.stages.toList.filter(_.isInstanceOf[XGBoostClassificationModel]).head.asInstanceOf[XGBoostClassificationModel]
model.nativeBooster.saveModel(xgb_model_scala_trained_model)`
**Predict ( python )**
`bst = xgb.Booster()`
`bst.load_model(xgb_model_scala_trained_model)`
`dtest = xgb.DMatrix(testDf)`
`bst.predict(dtest)`
In this case, the prediction probability is much different from the conventional one.
Below is a short summary.
Case1 : Python train , Python predict
Case2 : Spark train , Spark predcit
Case3 : Spark train, Python predict
case1 | case2 | case3
-- | -- | --
0.00240547 | 0.00238137 | 0.00857278
0.22821459 | 0.2271158 | 0.31658864
0.06961952 | 0.07109901 | 0.06855989
0.02959587 | 0.0273391 | 0.04815372
0.2269089 | 0.2301373 | 0.29909435
0.32883927 | 0.3200193 | 0.21665698
0.04287707 | 0.0412313 | 0.0999739
this is my parameters in the python
( Same option in the Scala )
params["objective"] = "binary:logistic"
params["eval_metric"] = "logloss"
params["eta"] = 0.016
params["min_child_weight"] = 1
params["gamma"] = 0.0365
params["subsample"] = 0.75
params["colsample_bytree"] = 0.77
params["scale_pos_weight"] = 1.0
params["max_depth"] = 9
params["tree_method"] = "hist"
I'm curious why there are different values for case3
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.