microsoft / microsoft/SynapseML

Bring your own Spark model

Open
#2,080 3 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

**Is your feature request related to a problem? Please describe.**
I'm wondering if it would be possible to "bring your own" Spark model for use with the interpretability functions, like `ICETransformer()`? For instance, we have several tools that allow us to do inference on Spark data frames by calling a `.predict()` or `.transform()` method. Is it possible to wrap such a non-PySpark MLLib model in a way that we could still use this package for generating explanations and ICE plots in Spark?

**Describe the solution you'd like**
Suppose I have a custom model (e.g., some kind of scoring code that operates on Spark data frames to transform the data by adding prediction columns). I'd like to wrap said model in a way that would allow me to call the `ICETransformer()` method. E.g.,

```python
pdp = ICETransformer(
model=custom_model, # custom_model.transform(spark_data_frame)
targetCol="prediction_col_name",
kind="average",
targetClasses=[1],
categoricalFeatures=categorical_features,
numericFeatures=numeric_features,
)
```

Where `custom_model` has a `.transform()` method similar to PySpark MLLib models that returns the input data with additional prediction columns.

Contributor guide

Open the contributing guide

Research direction

The issue names ICETransformer as the entry point and a custom model's transform method as the integration point; inspect how ICETransformer currently accepts and invokes models on Spark DataFrames. Define the supported wrapper contract so a non-PySpark model can generate explanations and ICE plots using targetCol, targetClasses, categoricalFeatures, and numericFeatures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.