dmlc / dmlc/xgboost

[jvm-packages] Issue in saving Xgboost model in spark scala and then load to the single Python model

Open
#4,765 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Hi.

I have trained XGBoost Classification model using scala ( spark 2.3 )
and then saved the model with nativeBooster.saveModel
( ref : https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html )

Then I loaded the model in Python3 and predict the probability

`spark env : spark 2.3 , xgb 0.82 `
`python env : python 3.6 , xgb 0.82 `

below is my shotly code

**Train ( Spark )**
`
val PIPELINE = new Pipeline().setStages(Array(VectorAssember_Input,XGBoostClassifier))
MDL = PIPELINE_666715c8_1acc_490f_bc46_e29f92070d23.fit(InputDataFrame)`

`MDL.write.overwrite().save(output_path)`
`val model = MDL.stages.toList.filter(_.isInstanceOf[XGBoostClassificationModel]).head.asInstanceOf[XGBoostClassificationModel]
model.nativeBooster.saveModel(xgb_model_scala_trained_model)`

**Predict ( python )**
`bst = xgb.Booster()`
`bst.load_model(xgb_model_scala_trained_model)`
`dtest = xgb.DMatrix(testDf)`
`bst.predict(dtest)`

In this case, the prediction probability is much different from the conventional one.
Below is a short summary.

Case1 : Python train , Python predict
Case2 : Spark train , Spark predcit
Case3 : Spark train, Python predict

case1 | case2 | case3
-- | -- | --
0.00240547 | 0.00238137 | 0.00857278
0.22821459 | 0.2271158 | 0.31658864
0.06961952 | 0.07109901 | 0.06855989
0.02959587 | 0.0273391 | 0.04815372
0.2269089 | 0.2301373 | 0.29909435
0.32883927 | 0.3200193 | 0.21665698
0.04287707 | 0.0412313 | 0.0999739

this is my parameters in the python
( Same option in the Scala )

params["objective"] = "binary:logistic"
params["eval_metric"] = "logloss"
params["eta"] = 0.016
params["min_child_weight"] = 1
params["gamma"] = 0.0365
params["subsample"] = 0.75
params["colsample_bytree"] = 0.77
params["scale_pos_weight"] = 1.0
params["max_depth"] = 9
params["tree_method"] = "hist"

I'm curious why there are different values ​​for case3

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.