microsoft / microsoft/SynapseML
TabularSHAP hangs indefinitely
@memoryz is already working on this.
Since May 1, 2023.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
Hi SynapseML team,
I am using TabularSHAP to explain predictions of the LightGBM classification model using SynapseML version `com.microsoft.azure:synapseml_2.12:0.9.5-35-e962330b-SNAPSHOT`
Based on the [tutorial](https://microsoft.github.io/SynapseML/docs/next/features/responsible_ai/Interpretability%20-%20Tabular%20SHAP%20explainer), I have prepared my code below. I am using only one executor with five cores to ensure the code doesn't fail due to concurrency, and I repartition all data frames to use a single partition.
```
from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler
from synapse.ml.lightgbm import JavaMLReadable, LightGBMClassifier, LightGBMClassificationModel
from synapse.ml.explainers import TabularSHAP
model = LightGBMClassifier.load('')
train_data = spark.read.parquet('').repartition(1).cache() # type: about 15K records
features = [...list of features...]
vector_assembler = VectorAssembler(inputCols = features, outputCol = 'features').setHandleInvalid("skip")
pipeline = Pipeline(stages=[vector_assembler, model])
model = pipeline.fit(train_data)
explain_instances = model.transform(train_data.limit(5).repartition(1).cache()).cache()
train_sample = train_data.orderBy(F.rand()).limit(100).repartition(1).cache()
shap = TabularSHAP(
inputCols=features,
outputCol="shapValues",
numSamples=10,
model=model,
targetCol="probability",
targetClasses=[1],
backgroundData=train_sample
)
explained = shap.transform(explain_instances).cache()
```
On the last line, when I call `shap.transform()`, the notebook shows me the progress bar but waits forever. I think it gets into some lock, but I am not sure. But I am sure it doesn't wait for the data frames to be calculated, because, while I have removed these lines from the code, I call `.show()` to force caching...
What am I missing?
Thank you.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.