combust / combust/mleap

Input schema of a StringIndexed Column is always string

Open
#761 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.5k
Forks
315
PR merge metrics
No merged PRs in 30d

Description

I have a dataset with a numerical column `rank` with values ranging from 1 to 5. When this dataset is read in spark with `inferSchema = true`, the dataType of the column is inferred as `int` by spark. A model is built by applying `StringIndexer` on the `rank` column as one of the stages and the `org.apache.spark.ml.PipelineModel` is exported as an mleap bundle. When we read the `ml.combust.mleap.runtime.frame.Transformer` from the mleap bundle and observe the `transformer.inputSchema`, it returns the `dataType` of the `rank` column as `String` instead of an `int`.

Looks like this is because of
https://github.com/combust/mleap/blob/master/mleap-core/src/main/scala/ml/combust/mleap/core/feature/StringIndexerModel.scala#L50

According to spark docs for [StringIndexer](https://spark.apache.org/docs/2.1.0/ml-features.html#stringindexer),
> If the input column is numeric, we cast it to string and index the string values.

so, an input column for a string indexer stage can be a numeric or string type.

Is there a way to pass the information of the actual datatype of the column to the transformer, so that `transformer.inputSchema` can return same instead of `String` for all the cases?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.