microsoft / microsoft/SynapseML
Bring your own Spark model
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
**Is your feature request related to a problem? Please describe.**
I'm wondering if it would be possible to "bring your own" Spark model for use with the interpretability functions, like `ICETransformer()`? For instance, we have several tools that allow us to do inference on Spark data frames by calling a `.predict()` or `.transform()` method. Is it possible to wrap such a non-PySpark MLLib model in a way that we could still use this package for generating explanations and ICE plots in Spark?
**Describe the solution you'd like**
Suppose I have a custom model (e.g., some kind of scoring code that operates on Spark data frames to transform the data by adding prediction columns). I'd like to wrap said model in a way that would allow me to call the `ICETransformer()` method. E.g.,
```python
pdp = ICETransformer(
model=custom_model, # custom_model.transform(spark_data_frame)
targetCol="prediction_col_name",
kind="average",
targetClasses=[1],
categoricalFeatures=categorical_features,
numericFeatures=numeric_features,
)
```
Where `custom_model` has a `.transform()` method similar to PySpark MLLib models that returns the input data with additional prediction columns.
Contributor guide
Research direction
The issue names ICETransformer as the entry point and a custom model's transform method as the integration point; inspect how ICETransformer currently accepts and invokes models on Spark DataFrames. Define the supported wrapper contract so a non-PySpark model can generate explanations and ICE plots using targetCol, targetClasses, categoricalFeatures, and numericFeatures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100