[jvm-package] Prediction contributions dropped for empty columns when training matrix is sparse
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
## Environment:
- XGBoost4J version: 0.82
- Spark Version: 2.3.3
- OpenJDK Version: 1.8.0_212
- OS: Ubuntu 18.04 LTS
## Steps to reproduce:
- Train a XGBoost model using a sparse input matrix (e.g. those constructed by `VectorAssembler`) with the XGBoost4J-Spark API;
- The sparse input matrix should contains some columns that are entirely empty, sometimes as a result of OneHotEncoderEstimator;
- Once the training is complete, invoke `setContribPredictionCol` on the trained model so that it produces the contributions of each feature per-prediction;
- Use the trained model to make predictions on any dataset;
- Observe the size of each contribution vector --- each contains less elements than expected (which is `number_of_features` + 1), depending on how many columns were empty in the training dataset;
## Minimal Example Reproducing the Issue:
1. From terminal, start a `spark-shell` in local mode with the `xgboost4j` dependency included:
```
spark-shell --packages ml.dmlc:xgboost4j-spark:0.82 --master local[4]
```
2. Once the "spark-shell" is ready, paste in the following code:
```scala
import org.apache.spark.sql.types.{DataTypes, StructField, StructType}
import org.apache.spark.sql.{Dataset, Row, SparkSession}
val raw_training_data = Seq(
Row(0.0, 1.0, 5.0, 1.0),
Row(1.0, 1.0, 5.0, 0.0),
Row(2.0, 1.0, 5.0, 1.0),
Row(3.0, 1.0, 5.0, 0.0),
Row(4.0, 1.0, 5.0, 1.0),
Row(5.0, 1.0, 5.0, 0.0)
)
val raw_testing_data = Seq(
Row(0.0, 1.0, 5.0, 1.0),
Row(1.0, 1.0, 4.0, 0.0),
Row(2.0, 1.0, 3.0, 1.0),
Row(3.0, 1.0, 2.0, 0.0),
Row(4.0, 1.0, 1.0, 1.0),
Row(5.0, 1.0, 0.0, 0.0)
)
val schema = new StructType(Array(
StructField("x1", DataTypes.DoubleType),
StructField("x2", DataTypes.DoubleType),
StructField("x3", DataTypes.DoubleType),
StructField("label", DataTypes.DoubleType)
))
val features = Array("x1", "x2", "x3")
val coded_features = features.map(f => "coded_" + f)
import org.apache.spark.ml.feature.{VectorAssembler, OneHotEncoderEstimator}
import spark.implicits._
val vectorAssembler = new VectorAssembler().setInputCols(coded_features).setOutputCol("features")
val oneHotEncoderEstimator = new OneHotEncoderEstimator().setHandleInvalid("error").setDropLast(true).setInputCols(features).setOutputCols(coded_features)
val oneHotTrainingDF = spark.createDataFrame(spark.sparkContext.parallelize( Seq( Row(Seq.fill(features.size)(5.0): _*) ) ) , StructType(features.map(f => StructField(f, DataTypes.DoubleType))))
val oneHotEncoder = oneHotEncoderEstimator.fit(oneHotTrainingDF);
val training_data = vectorAssembler.transform( oneHotEncoder.transform( spark.createDataFrame(spark.sparkContext.parallelize(raw_training_data), schema) ) )
val testing_data = vectorAssembler.transform( oneHotEncoder.transform( spark.createDataFrame(spark.sparkContext.parallelize(raw_testing_data), schema)) )
var xgbParam = Map(
"objective" -> "binary:logistic",
"num_round" -> 10,
"missing" -> java.lang.Double.NaN
)
import ml.dmlc.xgboost4j.scala.spark.{XGBoostClassifier, XGBoostClassificationModel}
val xgbClassifier: XGBoostClassifier = new XGBoostClassifier(xgbParam).setFeaturesCol("features").setLabelCol("label")
val xgbClassificationModel: XGBoostClassificationModel = xgbClassifier.fit(training_data)
xgbClassificationModel.setContribPredictionCol("contribPrediction")
val training_prediction: Dataset[Row] = xgbClassificationModel.transform(training_data)
training_prediction.withColumn("contrib_size", size( col("contribPrediction") ) ).show()
val testing_prediction: Dataset[Row] = xgbClassificationModel.transform(testing_data)
testing_prediction.withColumn("contrib_size", size( col("contribPrediction") ) ).show()
```
The output of the prediction on the training dataset:
```
+---+---+---+-----+-------------+-------------+---------+--------------------+--------------------+-------------+-----------+----------+------------+
| x1| x2| x3|label| coded_x1| coded_x2| coded_x3| features| contribPrediction|rawPrediction|probability|prediction|contrib_size|
+---+---+---+-----+-------------+-------------+---------+--------------------+--------------------+-------------+-----------+----------+------------+
|0.0|1.0|5.0| 1.0|(5,[0],[1.0])|(5,[1],[1.0])|(5,[],[])|(15,[0,6],[1.0,1.0])|[0.0, 0.0, 0.0, 0...| [-0.0,0.0]| [0.5,0.5]| 0.0| 8|
|1.0|1.0|5.0| 0.0|(5,[1],[1.0])|(5,[1],[1.0])|(5,[],[])|(15,[1,6],[1.0,1.0])|[0.0, 0.0, 0.0, 0...| [-0.0,0.0]| [0.5,0.5]| 0.0| 8|
|2.0|1.0|5.0| 1.0|(5,[2],[1.0])|(5,[1],[1.0])|(5,[],[])|(15,[2,6],[1.0,1.0])|[0.0, 0.0, 0.0, 0...| [-0.0,0.0]| [0.5,0.5]| 0.0| 8|
|3.0|1.0|5.0| 0.0|(5,[3],[1.0])|(5,[1],[1.0])|(5,[],[])|(15,[3,6],[1.0,1.0])|[0.0, 0.0, 0.0, 0...| [-0.0,0.0]| [0.5,0.5]| 0.0| 8|
|4.0|1.0|5.0| 1.0|(5,[4],[1.0])|(5,[1],[1.0])|(5,[],[])|(15,[4,6],[1.0,1.0])|[0.0, 0.0, 0.0, 0...| [-0.0,0.0]| [0.5,0.5]| 0.0| 8|
|5.0|1.0|5.0| 0.0| (5,[],[])|(5,[1],[1.0])|(5,[],[])| (15,[6],[1.0])|[0.0, 0.0, 0.0, 0...| [-0.0,0.0]| [0.5,0.5]| 0.0| 8|
+---+---+---+-----+-------------+-------------+---------+--------------------+--------------------+-------------+-----------+----------+------------+
```
One can see that the `contrib_size` column is 8; however, the input column (i.e. `features`) contains sparse vectors of size 15 --- the expected size of the contribution vector would be 15 + 1 = 16.
In the above example, the `OneHotEncoderEstimator` was trained not from the data, but separately to give it explicitly knowledge about the number of categories to expect (in this case, 5). The `x2` column in the training set contains only `1.0`s, so it is translated into a column of vectors containing only `[0, 1, 0, 0, 0]` in sparse format (`(5,[1],[1.0])`); The `x3` column in the training set contains only `5.0`s --- with the `dropLast` set to `True` on the `OneHotEncoderEstimator`, their values are translated into empty sparse vectors (`(5,[],[])`).
This issue does not seem to be present when the input matrix is dense.
## Why is this a problem?
It makes difficult for the consumers of the predictions to figure out what is the per-prediction contribution of each feature if the training data contains empty columns since the columns will be misaligned.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.