microsoft / microsoft/SynapseML
LightGBMClassifier: issue with featuresShapCol for multiclass model
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
Hello,
I train the multiclass classification model.
Target variable consists of 5 classes.
**Problem**: when I call shape values, I obtain 1D vector .
The code that I am running on pyspark is:
```
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName("My app") \
.config("spark.jars", "postgresql-42.4.2.jar") \
.config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
.config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT") \
.getOrCreate()
# -> spark version: 3.3.0
from pyspark.ml.feature import OneHotEncoder, VectorAssembler, StringIndexer
from pyspark.ml import Pipeline, PipelineModel
from pyspark.sql import Row
from synapse.ml.lightgbm import LightGBMClassifier
features ==> list of features
ohe_features ==> list of categorical features for one-hot encoding
ohe_reordered_features = [f'category_mapped_{it}' for it in ohe_features]
preOheEncoder = StringIndexer(inputCols=ohe_features, outputCols=ohe_reordered_features, handleInvalid='keep')
ohe_output_features = [f'ohe_out_{it}' for it in ohe_features]
oheEncoder = OneHotEncoder(inputCols=ohe_reordered_features, outputCols=ohe_output_features, dropLast=False)
vecAssembler = VectorAssembler(inputCols=[*features, *ohe_output_features], outputCol="features", handleInvalid="keep")
model = LightGBMClassifier(featuresCol='features', labelCol='target', rawPredictionCol='rawPrediction',
featuresShapCol='featuresShap', probabilityCol='probability', weightCol='weight',
numLeaves=4, numIterations=25,
objective='multiclass',
boostingType='gbdt'
)
data_pipeline = Pipeline(stages=[preOheEncoder, oheEncoder, vecAssembler])
train_data = data_pipeline.transform(train_data)
test_data = data_pipeline.transform(test_data)
model = model.fit(train_data)
preds = model.transform(test_data)
```
When I select shape values I get 1D array, instead of 5D array (as I have 5 classes in target):
```
# Get shap values:
shap_values = preds.select("featuresShap")
# transform dataframe with pyspark VectorUDT to numpy array
shap_values_np = np.array(shap_values.select(vector_to_array("featuresShap").alias("featuresShap")).limit(100).collect())
print(shap_values_np.shape)
# -> (100, 1, 7010)
# removing unnecessary axis
shap_values_np_3 = np.squeeze(shap_values_np, axis=1)
print(shap_values_np_3.shape)
# -> (100, 7010)
Number of features in 'features' column:
shap_values_pd = shap_values.select(
[vector_to_array("featuresShap").alias("featuresShap"),\
vector_to_array("features").alias("features")])
.limit(100).toPandas()
print(len(shap_values_pd.features[0]))
# -> 1402
```
Why am I getting a 1D vector for shap values even though the classification is multiclass?
Is it because of any bug in my code, or this is as it should be? and then how to interpret correctly such 1D vector with shap values in case of multiclass classification?
Contributor guide
Assessment
This issue has not been assessed yet.