combust / combust/mleap

MLeap BinaryLogisticRegressionModel calculating result differs with the Spark model

Open
#339 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

Hi there. I tried to train a Spark `BinaryLogisticRegressionModel` with a dataset whose labels are the same value and used this model to make predictions.

``` Scala
// data
val rddData = sc.parallelize(Seq[(Integer, Double, Double, Double, Double, Double, Double, Double, Double, Double, Double, Double, Double, Double, Double)](
(1, 0.2, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4),
(1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.8, 0.4, 0.2, 0.1, 1.2, 1.1, 1.1, 1.0, 0.33),
(1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.8, 0.4, 0.2, 0.1, 1.2, 1.1, 1.1, 1.0, 0.33)))
val data = spark.createDataFrame(rddData).toDF("LABEL", "C1", "C2", "C3", "C4", "C5", "C6", "C7", "C8",
"C9", "C10", "C11", "C12", "C13", "C14")

// transformers & estimators
val assembler = new VectorAssembler().setInputCols(Array("C1", "C2", "C3", "C4", "C5", "C6", "C7", "C8", "C9", "C10", "C11", "C12", "C13", "C14")).setOutputCol("features")
val featureIndexer = new VectorIndexer().setInputCol("features").setOutputCol("indexedFeatures").setMaxCategories(2)
val lr = new LogisticRegression().setLabelCol("LABEL")
```

The result looks fine:

``` JSON
{
"probability":[0.0,1.0],
"prediction":1.0
}
```

After converting to MLeap model, the "probabilities" are all nulls, the prediction result is incorrect as well.

``` JSON
{
"probability": [null, null],
"prediction": 0.0
}
```

Spark Version: 2.1.1
MLeap Version: 0.7.0

Seems that Spark set the `intercept` parameter to `Double.PositiveInfinity` but MLeap can't handle this situation.

``` Scala
// org.apache.spark.ml.classification.LogisticRegression
val interceptVec = if (isMultinomial) {
Vectors.sparse(numClasses, Seq((constantLabelIndex, Double.PositiveInfinity)))
} else {
Vectors.dense(if (numClasses == 2) Double.PositiveInfinity else Double.NegativeInfinity)
}
```

``` scala
// ml.combust.mleap.core.classification.BinaryLogisticRegressionModel
def margin(features: Vector): Double = {
BLAS.dot(features, coefficients) + intercept
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.