combust / combust/mleap

Impossible to deserialize a bundle written with Scikit-Learn with Pyspark: no "bundle.json found"

Open
#691 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

I serialize a model with Scikit-Learn:

```
#Generate data
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(100, 5), columns=['a', 'b', 'c', 'd', 'e'])
df["y"] = (df['a'] > 0.5).astype(int)
df.head()

from mleap.sklearn.ensemble.forest import RandomForestClassifier

forestModel = RandomForestClassifier()
forestModel.mlinit(input_features='a',
feature_names='a',
prediction_column='e_binary')

forestModel.fit(df[['a']], df[['y']])

forestModel.serialize_to_bundle("/dbfs/FileStore/tables/mleaptestmodelforest", "model.json")
```
When I try to read it with Pyspark:

```
from pyspark.ml.classification import RandomForestClassificationModel

model = RandomForestClassificationModel.deserializeFromBundle("file:/dbfs/FileStore/tables/mleaptestmodelforest")
```

I have this error:
`java.nio.file.NoSuchFileException: /dbfs/FileStore/tables/mleaptestmodelforest/bundle.json`

I have no "bundle.json".

Could you help me please?
Is it really possible to seralize a model with Scikit-Learn and deserialize it with Pyspark?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the scikit-learn serialize_to_bundle call and the PySpark RandomForestClassificationModel.deserializeFromBundle entry point shown in the reproduction. Trace the expected bundle layout and determine whether the generated model can be consumed by the Spark reader; done means the compatibility behavior is verified and the missing bundle.json failure is resolved or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.