microsoft / microsoft/SynapseML

LightGBMClassifier: issue with featuresShapCol for multiclass model

Open
#1,669 2 comments 0 reactions 1 assignee Assigned to @imatiach-msft View on GitHub
area/lightgbm bug
Dominant language
Scala
Stars
5.2k
Forks
868
Avg merge
22h 9m
Merged PRs (30d)
45

Description

Hello,
I train the multiclass classification model.
Target variable consists of 5 classes.
**Problem**: when I call shape values, I obtain 1D vector .
The code that I am running on pyspark is:
```
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName("My app") \
.config("spark.jars", "postgresql-42.4.2.jar") \
.config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
.config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT") \
.getOrCreate()

# -> spark version: 3.3.0

from pyspark.ml.feature import OneHotEncoder, VectorAssembler, StringIndexer
from pyspark.ml import Pipeline, PipelineModel
from pyspark.sql import Row
from synapse.ml.lightgbm import LightGBMClassifier

features ==> list of features
ohe_features ==> list of categorical features for one-hot encoding

ohe_reordered_features = [f'category_mapped_{it}' for it in ohe_features]
preOheEncoder = StringIndexer(inputCols=ohe_features, outputCols=ohe_reordered_features, handleInvalid='keep')

ohe_output_features = [f'ohe_out_{it}' for it in ohe_features]
oheEncoder = OneHotEncoder(inputCols=ohe_reordered_features, outputCols=ohe_output_features, dropLast=False)

vecAssembler = VectorAssembler(inputCols=[*features, *ohe_output_features], outputCol="features", handleInvalid="keep")

model = LightGBMClassifier(featuresCol='features', labelCol='target', rawPredictionCol='rawPrediction',
featuresShapCol='featuresShap', probabilityCol='probability', weightCol='weight',
numLeaves=4, numIterations=25,
objective='multiclass',
boostingType='gbdt'
)
data_pipeline = Pipeline(stages=[preOheEncoder, oheEncoder, vecAssembler])
train_data = data_pipeline.transform(train_data)
test_data = data_pipeline.transform(test_data)
model = model.fit(train_data)
preds = model.transform(test_data)
```
When I select shape values I get 1D array, instead of 5D array (as I have 5 classes in target):

```
# Get shap values:
shap_values = preds.select("featuresShap")

# transform dataframe with pyspark VectorUDT to numpy array
shap_values_np = np.array(shap_values.select(vector_to_array("featuresShap").alias("featuresShap")).limit(100).collect())
print(shap_values_np.shape)
# -> (100, 1, 7010)
# removing unnecessary axis
shap_values_np_3 = np.squeeze(shap_values_np, axis=1)
print(shap_values_np_3.shape)
# -> (100, 7010)

Number of features in 'features' column:
shap_values_pd = shap_values.select(
[vector_to_array("featuresShap").alias("featuresShap"),\
vector_to_array("features").alias("features")])
.limit(100).toPandas()
print(len(shap_values_pd.features[0]))
# -> 1402

```
Why am I getting a 1D vector for shap values even though the classification is multiclass?
Is it because of any bug in my code, or this is as it should be? and then how to interpret correctly such 1D vector with shap values in case of multiclass classification?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.