combust / combust/mleap

org.apache.spark.sql.mleap.TypeConverters can not convert 2D tensor to Matrix

Open
#854 0 comments 0 reactions 1 assignee Claimed by @austinzh View on GitHub
bug
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

Current implementation always will convert Tensor to Vector
Bug is hidden in `tt.dimensions.size` where `tt.dimensions` is `Option[Seq[Int]]`, so calling `size` on Some will have size of 1 and calling `size` on None will have size of 0. So in following code, `TensorType` will always convert to `VectorUDT`
```
def mleapTensorToSpark(tt: types.TensorType): DataType = {
assert(TypeConverters.VECTOR_BASIC_TYPES.contains(tt.base),
s"cannot convert tensor with base ${tt.base} to vector")
assert(tt.dimensions.isDefined, "cannot convert tensor with undefined dimensions")

if(tt.dimensions.isEmpty) {
mleapBasicTypeToSparkType(tt.base)
} else if(tt.dimensions.size == 1) {
new VectorUDT
} else if(tt.dimensions.size == 2) {
new MatrixUDT
} else {
throw new IllegalArgumentException("cannot convert tensor for non-scalar, vector or matrix tensor")
}
}
```
Same bug exists in `mleapToSparkValue` function as well.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.