Impossible to deserialize a bundle written with Scikit-Learn with Pyspark: no "bundle.json found"
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 315
- PR merge metrics
- No merged PRs in 30d
Description
I serialize a model with Scikit-Learn:
```
#Generate data
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(100, 5), columns=['a', 'b', 'c', 'd', 'e'])
df["y"] = (df['a'] > 0.5).astype(int)
df.head()
from mleap.sklearn.ensemble.forest import RandomForestClassifier
forestModel = RandomForestClassifier()
forestModel.mlinit(input_features='a',
feature_names='a',
prediction_column='e_binary')
forestModel.fit(df[['a']], df[['y']])
forestModel.serialize_to_bundle("/dbfs/FileStore/tables/mleaptestmodelforest", "model.json")
```
When I try to read it with Pyspark:
```
from pyspark.ml.classification import RandomForestClassificationModel
model = RandomForestClassificationModel.deserializeFromBundle("file:/dbfs/FileStore/tables/mleaptestmodelforest")
```
I have this error:
`java.nio.file.NoSuchFileException: /dbfs/FileStore/tables/mleaptestmodelforest/bundle.json`
I have no "bundle.json".
Could you help me please?
Is it really possible to seralize a model with Scikit-Learn and deserialize it with Pyspark?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the scikit-learn serialize_to_bundle call and the PySpark RandomForestClassificationModel.deserializeFromBundle entry point shown in the reproduction. Trace the expected bundle layout and determine whether the generated model can be consumed by the Spark reader; done means the compatibility behavior is verified and the missing bundle.json failure is resolved or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100