microsoft / microsoft/SynapseML

Cannot load PipelineModel

Open
#614 24 comments 0 reactions 1 assignee View on GitHub

@imatiach-msft is already working on this.

Since Jul 13, 2019.

area/lightgbm bug high priority
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

Hi,

I am trying to port my ML pipeline so I can use LightGBM instead of the PySpark GBT. I have been able to design a Pipeline with a LightGBM as final estimator. Once trained, I save the PipelineModel object to disk succesfully.

Problem is, when I want to load the model again to evaluate it, the following error appears:

```
2019-07-11 10:44:03 INFO DAGScheduler:54 - Job 66 finished: runJob at PythonRDD
.scala:152, took 0,709961 s
Traceback (most recent call last):
File "C:/Users/Y0644483/Documents/Workspace/ninabrlong/bin/eval_model.py", lin
e 86, in
model = ml.PipelineModel.load(args[""])
File "C:\Users\Y0644483\AppData\Local\Continuum\miniconda3\envs\ninabrlong\lib
\site-packages\pyspark\python\lib\pyspark.zip\pyspark\ml\util.py", line 311, in
load
File "C:\Users\Y0644483\AppData\Local\Continuum\miniconda3\envs\ninabrlong\lib
\site-packages\pyspark\python\lib\pyspark.zip\pyspark\ml\pipeline.py", line 244,
in load
File "C:\Users\Y0644483\AppData\Local\Continuum\miniconda3\envs\ninabrlong\lib
\site-packages\pyspark\python\lib\pyspark.zip\pyspark\ml\pipeline.py", line 378,
in load
File "C:\Users\Y0644483\AppData\Local\Continuum\miniconda3\envs\ninabrlong\lib
\site-packages\pyspark\python\lib\pyspark.zip\pyspark\ml\util.py", line 535, in
loadParamsInstance
File "C:\Users\Y0644483\AppData\Local\Continuum\miniconda3\envs\ninabrlong\lib
\site-packages\pyspark\python\lib\pyspark.zip\pyspark\ml\util.py", line 478, in
__get_class
AttributeError: module 'com.microsoft.ml.spark' has no attribute 'LightGBMRegres
sionModel'
```

I could not find any reference to this error, and I do not have a clue on what it could be happening. Besides, I found some references in your docs about using saveNativeModel(), but do not know how that fits in a whole-pipeline-saving scenario.

I am using mmlspark 0.17 and pyspark 2.3.2 in standalone mode in my local development environment.

I looked into the saved model file and found the following structure:

```
{"class":"pyspark.ml.pipeline.PipelineModel","timestamp":1562834309828,"sparkVersion":"2.3.2","uid":"PipelineModel_423e9b309dc390188fb9","paramMap":{"stageUids":["CategoricalImputerModel_44e1b6199ae304e52301","Imputer_4dd2932c4e613d1a22a7","VectorAssembler_4b84b526562e9c57d94b","StandardScaler_435a845ad25d209ac500","StringIndexer_43adbca01f7d9b98b4a4","StringIndexer_44adb088b5df936619a3","StringIndexer_4f47ae3f303a64b83a33","StringIndexer_466ea94e036991e2b49c","StringIndexer_4e25a7fd976a2cd42a2d","StringIndexer_42a180d928833d6d08ba","StringIndexer_4544901887ec85bf8f93","StringIndexer_410c9fae53c67291e238","StringIndexer_48c5a6c27b7029672329","StringIndexer_4faabb0736b77c4e2e2d","StringIndexer_438795bd74a5ec9f9d8e","StringIndexer_416d809ec7e5c7a7ad58","StringIndexer_4c9b847fc6c2ed13b53a","VectorAssembler_45978399a1e581608699","LightGBMRegressionModel_4c6d84e3292c452f4ce5"],"language":"Python"}}
```

Any hint or help would be much appreciated.

Regards,
Gus.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.