combust / combust/mleap

Serialized then deserialized model does not have the same results (all zero's)

Open
#530 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

It's not clear to me why this is happening. I was first using spark 2.4.3 but then changed back to 2.3.3 (as I believe support for 2.4 was being considered and may have just been added). I have attached the data file (public domain). I also checked that the GBTClassifier is supported from Spark ML.

Python version 3.7.3
mleap version 0.13.0

```
import pandas as pd
import pyspark
from pyspark.ml import Pipeline
from pyspark.ml import PipelineModel
from pyspark.ml.recommendation import ALS
from pyspark.ml.feature import OneHotEncoderEstimator, StringIndexer, VectorAssembler
from pyspark.ml.classification import GBTClassifier
import mleap
import mleap.pyspark
from mleap.pyspark.spark_support import SimpleSparkSerializer # this causes the dynamic binding to model

dataset_location = "/income.data.4k.txt"
models_folder = ""

df = pd.read_csv(dataset_location)
df.columns = [col.strip() for col in df.columns]
for col in df.columns:
if str(df.dtypes[col]) == "object":
df[col] = df[col].apply(lambda x: x.strip() if type(x) is str else x)

df = pd.get_dummies(df)

df = df.drop("yearly-income_<=50K", axis = 1)
target = "yearly-income"
columns = list(df.columns)
columns[columns.index("yearly-income_>50K")] = target
df.columns = columns
sqlCtx = SQLContext(SparkContext.getOrCreate())
sdf = sqlCtx.createDataFrame(df)

assembler = VectorAssembler(inputCols=[col for col in columns if col != target], outputCol="features")
gbt = GBTClassifier(maxIter=10, maxDepth=4, labelCol = target, featuresCol = "features")
pipeline = Pipeline(stages=[assembler, gbt])
model = pipeline.fit(sdf)
results = model.transform(sdf)
# FIXME: Not sure why you need the df after. It fails if I dont supply it.
# path must be absolute
r = results.toPandas()
r.describe()
r.head()
model.serializeToBundle("jar:file:{}/pyspark.gbt_pipeline.zip".format(models_folder), results)
model = PipelineModel.deserializeFromBundle("jar:file:{}/pyspark.gbt_pipeline.zip".format(models_folder))

results2 = model.transform(sdf)
r2 = results2.toPandas()
r2.describe()
r2.head()
```

The output from the first time:

```
>>> r.describe()
age fnlwgt education-num capital-gain ... native-country_Vietnam native-country_Yugoslavia yearly-income prediction
count 4000.000000 4.000000e+03 4000.00000 4000.000000 ... 4000.000000 4000.000000 4000.000000 4000.000000
mean 38.213750 1.889953e+05 10.07600 1071.823500 ... 0.002500 0.000500 0.242750 0.202000
std 13.764971 1.070024e+05 2.58629 7294.042604 ... 0.049944 0.022358 0.428799 0.401542
min 17.000000 1.228500e+04 1.00000 0.000000 ... 0.000000 0.000000 0.000000 0.000000
25% 27.000000 1.169508e+05 9.00000 0.000000 ... 0.000000 0.000000 0.000000 0.000000
50% 36.000000 1.771715e+05 10.00000 0.000000 ... 0.000000 0.000000 0.000000 0.000000
75% 47.000000 2.366330e+05 13.00000 0.000000 ... 0.000000 0.000000 0.000000 0.000000
max 90.000000 1.226583e+06 16.00000 99999.000000 ... 1.000000 1.000000 1.000000 1.000000

[8 rows x 108 columns]
>>> r.head()
age fnlwgt education-num ... rawPrediction probability prediction
0 53 20438 10 ... [1.1557390749607264, -1.1557390749607264] [0.9098232035096955, 0.09017679649030452] 0.0
1 39 216552 14 ... [-0.34625038604211694, 0.34625038604211694] [0.3334769951336344, 0.6665230048663656] 1.0
2 54 143865 9 ... [1.1837202808156024, -1.1837202808156024] [0.9143105493591535, 0.08568945064084654] 0.0
3 38 172855 13 ... [-1.4172112252841806, 1.4172112252841806] [0.055492148863025934, 0.9445078511369741] 1.0
4 17 198146 7 ... [1.3097197675583097, -1.3097197675583097] [0.932102244506494, 0.06789775549350596] 0.0

```

The second time (scroll to the right to see that all the predictions are 0):

```
>>> r2.describe()
age fnlwgt education-num capital-gain ... native-country_Vietnam native-country_Yugoslavia yearly-income prediction
count 4000.000000 4.000000e+03 4000.00000 4000.000000 ... 4000.000000 4000.000000 4000.000000 4000.0
mean 38.213750 1.889953e+05 10.07600 1071.823500 ... 0.002500 0.000500 0.242750 0.0
std 13.764971 1.070024e+05 2.58629 7294.042604 ... 0.049944 0.022358 0.428799 0.0
min 17.000000 1.228500e+04 1.00000 0.000000 ... 0.000000 0.000000 0.000000 0.0
25% 27.000000 1.169508e+05 9.00000 0.000000 ... 0.000000 0.000000 0.000000 0.0
50% 36.000000 1.771715e+05 10.00000 0.000000 ... 0.000000 0.000000 0.000000 0.0
75% 47.000000 2.366330e+05 13.00000 0.000000 ... 0.000000 0.000000 0.000000 0.0
max 90.000000 1.226583e+06 16.00000 99999.000000 ... 1.000000 1.000000 1.000000 0.0

[8 rows x 108 columns]
>>> r2.head()
age fnlwgt education-num capital-gain capital-loss ... yearly-income features rawPrediction probability prediction
0 53 20438 10 0 0 ... 0 (53.0, 20438.0, 10.0, 0.0, 0.0, 15.0, 0.0, 0.0... [-0.0, 0.0] [0.5, 0.5] 0.0
1 39 216552 14 0 0 ... 0 (39.0, 216552.0, 14.0, 0.0, 0.0, 40.0, 0.0, 0.... [-0.0, 0.0] [0.5, 0.5] 0.0
2 54 143865 9 0 0 ... 0 (54.0, 143865.0, 9.0, 0.0, 0.0, 35.0, 0.0, 0.0... [-0.0, 0.0] [0.5, 0.5] 0.0
3 38 172855 13 0 1887 ... 1 (38.0, 172855.0, 13.0, 0.0, 1887.0, 40.0, 0.0,... [-0.0, 0.0] [0.5, 0.5] 0.0
4 17 198146 7 0 0 ... 0 (17.0, 198146.0, 7.0, 0.0, 0.0, 16.0, 0.0, 0.0... [-0.0, 0.0] [0.5, 0.5] 0.0

```
[income.data.4k.txt](https://github.com/combust/mleap/files/3244289/income.data.4k.txt)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the serializeToBundle and deserializeFromBundle calls around the PipelineModel, comparing transform results before and after persistence for the GBTClassifier pipeline. Reproduce the all-zero predictions with the supplied income data and verify that the deserialized model returns predictions and probabilities matching the original model.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.